11-22-2001
remove unnecessary lines
If I have a file with 5000 lines contains numbers. Some of the number are repeated and some are not. Among those repeated number, I only would like to keep only one. How do I remove those balance repeated number.
Your help is much appreciated. Thank you.
10 More Discussions You Might Find Interesting
1. IP Networking
Good day :)
I recently checked some stuff on my gateway and discovered what I believe is an unneeded route entry.
# route -n
Kernel IP routing table
Destination Gateway Genmask ... Iface
287.265.45.0 0.0.0.0 255.255.255.0 ... eth0
192.168.0.0 0.0.0.0 ... (3 Replies)
Discussion started by: J.P
3 Replies
2. UNIX for Dummies Questions & Answers
hi folks...
i have to write a sript that removes unnecessary backup-files.
iam new to shell scripting so please be patient with me. and no its not homework :p
these files look like "javacore303330.1209029863.txt" where the first number is the PID and the second is the timestamp. so there can be... (5 Replies)
Discussion started by: cypher82
5 Replies
3. UNIX for Advanced & Expert Users
HI all,
I have a file with following data - test1
"ABC,D",1234,"XYZ,QWER",1234
"SZXA",9870,"ASD,QWERT",234
"XZ,SD",9478,"ADCS,AXZ",876
"WESR",8764,"AQZXAS",9888
"WESR",9898,"WESDRTSAW",3323
I need to get rid of unnecessary commas in fields having double quotes.
Ouput -
... (1 Reply)
Discussion started by: sumeet
1 Replies
4. Shell Programming and Scripting
Hi,
I'm not a expert in shell programming, so i've come here to take help from u gurus.
I'm trying to tailor a csv file that i got to make it work for the LOAD FROM command.
I've a datatable csv of the below format -
--in file format
xx,xx,xx ,xx , , , , ,,xx,
xxxx,, ,, xxx,... (11 Replies)
Discussion started by: dvah
11 Replies
5. Solaris
I want to clean used solaris OS and then to give another developer.
How can I understand that difference between system file/folder and others.
I want to delete apart from the files/folders.
I need tools or scripts like disk-cleanup or something like that.
#usr>du -s -h *
6.6M 4lib
... (1 Reply)
Discussion started by: getrue
1 Replies
6. UNIX for Dummies Questions & Answers
Hi,
I have a huge file which has Lacs of lines. File system got full.
I want your guys help to suggest me a solution so that I can remove all lines from that file but not last 50,000 lines. I want solution which can remove lines from existing file so that I can have some space left with. (28 Replies)
Discussion started by: prashant2507198
28 Replies
7. Shell Programming and Scripting
I have two files, a keepout.txt and a database.csv. They're unsorted, but could be sorted.
keepout:
user1
buser3
anuser19
notheruser27
database:
user1,2343,"information about",field,blah,34
user2,4231,"mo info",etc,stuff,43
notheruser27,4344,"hiya",thing,more thing,423... (4 Replies)
Discussion started by: esoffron
4 Replies
8. Shell Programming and Scripting
i have data as below
123,"paul phiri",paul@yahoo.com,"po.box 23, BT","Eco Bank,Blantyre,Malawi"
i need an output to be
123,"paul phiri",paul@yahoo.com,"po.box 23 BT","Eco Bank Blantyre Malawi" (5 Replies)
Discussion started by: mathias23
5 Replies
9. Shell Programming and Scripting
Hello everyone,
Although it seems easy, I've been stuck with this problem for a moment now and I can't figure out a way to get it done.
My problem is the following:
I have a file where each line is a sequence of IP addresses, example :
10.0.0.1 10.0.0.2
10.0.0.5 10.0.0.1 10.0.0.2... (5 Replies)
Discussion started by: MisterJellyBean
5 Replies
10. Shell Programming and Scripting
I have been searching and trying to come up with an awk that will perform the following on a
converted text file (original is a pdf).
1. Since the first two lines are (begin with) text they are removed
2. if $1 is a number then all text is merged (combined) into one line until the next... (3 Replies)
Discussion started by: cmccabe
3 Replies
uniq(1) General Commands Manual uniq(1)
NAME
uniq - report repeated lines in a file
SYNOPSIS
fields] chars] [input_file [output_file]]
DESCRIPTION
reads the input text file input_file, comparing adjacent lines, and copies the result to output_file. If input_file is not specified, the
standard input and standard output are used. If input_file is specified, but output_file is not, results are printed to standard output.
input_file and output_file must not be the same file.
Line-Comparison Options
recognizes the following options when comparing adjacent lines:
Print those lines that are repeated in the original file.
Print copy only of each repeated line in the input file.
Generate an output report in default style
except that each line is preceded by a count of the number of times it occurred. If this option is specified, the and
options are ignored if either or both are also present.
If none of the options or are present, prints the results of the union of the and options, producing a copy of the original input file with
the second and succeeding copies of any repeated lines removed. (Note that repeated lines must be adjacent in order to be found -- see
sort(1)).
Field-Skip Options
Two options are provided for skipping an initial portion of each line when making comparisons:
Ignore the first
fields fields, together with any blanks before each. fields is a positive decimal integer. A field is defined as a
string of non-space, non-tab characters separated by tabs and/or spaces from its neighbors.
Ignore the first
chars characters. chars is a positive decimal integer. Each line in the input is assumed to be terminated with a
new line character for purposes of comparison. Fields are skipped before characters.
EXTERNAL INFLUENCES
Environment Variables
must be equal to the value it had when the input files were sorted.
determines the interpretation of text within files as single- and/or multi-byte characters, and defines a space character when the or
option is used.
determines the language in which messages are displayed.
If or is not specified in the environment or is set to the empty string, the value of is used as a default for each unspecified or empty
variable. If is not specified or is set to the empty string, a default of "C" (see lang(5)) is used instead of If any internationalization
variable contains an invalid setting, behaves as if all internationalization variables are set to "C". See environ(5).
International Code Set Support
Single- and multi-byte character code sets are supported.
RETURN VALUE
Exit values are:
0 Successful completion.
>0 Error condition occurred.
AUTHOR
was developed by OSF and HP.
SEE ALSO
comm(1), sort(1).
STANDARDS CONFORMANCE
uniq(1)