10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi, all
I have a csv file that I would like to remove duplicate lines based on 1st field and sort them by the 1st field. If there are more than 1 line which is same on the 1st field, I want to keep the first line of them and remove the rest. I think I have to use uniq or something, but I still... (8 Replies)
Discussion started by: refrain
8 Replies
2. Shell Programming and Scripting
Dear community,
I have to remove duplicate lines from a file contains a very big ammount of rows (milions?) based on 1st and 3rd columns
The data are like this:
Region 23/11/2014 09:11:36 41752
Medio 23/11/2014 03:11:38 4132
Info 23/11/2014 05:11:09 4323... (2 Replies)
Discussion started by: Lord Spectre
2 Replies
3. Shell Programming and Scripting
hi,
Please help me to write a command to delete duplicate lines from a file. And the size of file is 50 MB. How to remove duplicate lins from such a big file. (6 Replies)
Discussion started by: vsachan
6 Replies
4. Shell Programming and Scripting
Hey guys, need some help to fix this script. I am trying to remove all the duplicate lines in this file.
I wrote the following script, but does not work. What is the problem?
The output file should only contain five lines:
Later! (5 Replies)
Discussion started by: Ernst
5 Replies
5. Shell Programming and Scripting
Hi,
I have two files with below data::
file1:-
123|aaa|ppp
445|fff|yyy
999|ttt|jjj
555|hhh|hhh
file2:-
445|fff|yyy
555|hhh|hhh
The records present in file1, not present in file 2 should be writtent to the out put file.
output:-
123|aaa|ppp
999|ttt|jjj
Is there any one line... (3 Replies)
Discussion started by: gani_85
3 Replies
6. Shell Programming and Scripting
greetings,
i'm hoping there is a way to cat a file, remove duplicate lines and send that output to a new file. the file will always vary but be something similar to this:
please keep in mind that the above could be eight occurrences of each hostname or it might simply have another four of an... (2 Replies)
Discussion started by: crimso
2 Replies
7. UNIX for Dummies Questions & Answers
Hi please help me how to remove duplicate lines in any file.
I have a file having huge number of lines.
i want to remove selected lines in it.
And also if there exists duplicate lines, I want to delete the rest & just keep one of them.
Please help me with any unix commands or even fortran... (7 Replies)
Discussion started by: reva
7 Replies
8. Shell Programming and Scripting
Hello,
Can anyone tell Command/Script to remove duplicate lines from the file? (2 Replies)
Discussion started by: Rahulpict
2 Replies
9. UNIX for Dummies Questions & Answers
I have a log file "logreport" that contains several lines as seen below:
04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping
06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping
07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but... (18 Replies)
Discussion started by: Nysif Steve
18 Replies
10. Shell Programming and Scripting
I am doing KSH script to remove duplicate lines in a file. Let say the file has format below.
FileA
1253-6856
3101-4011
1827-1356
1822-1157
1822-1157
1000-1410
1000-1410
1822-1231
1822-1231
3101-4011
1822-1157
1822-1231
and I want to simply it with no duplicate line as file... (5 Replies)
Discussion started by: Teh Tiack Ein
5 Replies
GIT-STRIPSPACE(1) Git Manual GIT-STRIPSPACE(1)
NAME
git-stripspace - Remove unnecessary whitespace
SYNOPSIS
git stripspace [-s | --strip-comments]
git stripspace [-c | --comment-lines]
DESCRIPTION
Read text, such as commit messages, notes, tags and branch descriptions, from the standard input and clean it in the manner used by Git.
With no arguments, this will:
o remove trailing whitespace from all lines
o collapse multiple consecutive empty lines into one empty line
o remove empty lines from the beginning and end of the input
o add a missing
to the last line if necessary.
In the case where the input consists entirely of whitespace characters, no output will be produced.
NOTE: This is intended for cleaning metadata, prefer the --whitespace=fix mode of git-apply(1) for correcting whitespace of patches or
files in the repository.
OPTIONS
-s, --strip-comments
Skip and remove all lines starting with comment character (default #).
-c, --comment-lines
Prepend comment character and blank to each line. Lines will automatically be terminated with a newline. On empty lines, only the
comment character will be prepended.
EXAMPLES
Given the following noisy input with $ indicating the end of a line:
|A brief introduction $
| $
|$
|A new paragraph$
|# with a commented-out line $
|explaining lots of stuff.$
|$
|# An old paragraph, also commented-out. $
| $
|The end.$
| $
Use git stripspace with no arguments to obtain:
|A brief introduction$
|$
|A new paragraph$
|# with a commented-out line$
|explaining lots of stuff.$
|$
|# An old paragraph, also commented-out.$
|$
|The end.$
Use git stripspace --strip-comments to obtain:
|A brief introduction$
|$
|A new paragraph$
|explaining lots of stuff.$
|$
|The end.$
GIT
Part of the git(1) suite
Git 2.17.1 10/05/2018 GIT-STRIPSPACE(1)