08-22-2008
extracting unique lines from text file
I have a file with 14million lines and I would like to extract all the unique lines from the file into another text file.
For example:
Contents of file1
happy
sad
smile
happy
funny
sad
I want to run a command against file one that only returns the unique lines (ie 1 line for happy and 1 line for sad).
Could someone please point me in the right direction.
Thanks
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Greetings,
I would like to extract records from a fixed width text file that have unique field elements.
Data is structured like this:
John A Smith NY
Mary C Jones WA
Adam J Clark PA
Mary Jones WA
Fieldname / start-end position
Firstname 1-10... (8 Replies)
Discussion started by: sitney
8 Replies
2. Shell Programming and Scripting
Hi all! Im trying to extract a portion of text from a file and put it into a new file. I need all the lines between <Placement> and </Placement> including the Placemark lines themselves. Is there a way to extract all instances of these and not just the first one found? I've tried using sed and... (4 Replies)
Discussion started by: Grizzly
4 Replies
3. Shell Programming and Scripting
Hi,
snp.txt
CHR_A SNP_A BP_A_st BP_A_End CHR_B BP_B SNP_B R2 p-SNP_A p-SNP_B
5 rs1988728 74904317 74904318 5 74960646 rs1427924 0.377333 0.000740085 0.013930081
5 ... (12 Replies)
Discussion started by: genehunter
12 Replies
4. Shell Programming and Scripting
I'm attempting to write a script to identify users who have sudo access on a server. I only want to extract the ID's of the sudo users after a unique line of text. The list of sudo users goes to the EOF so I only need the script to start after the unique line of text. I already have a script to... (1 Reply)
Discussion started by: bouncer
1 Replies
5. UNIX for Advanced & Expert Users
Hi All,
I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space.
I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies
6. Shell Programming and Scripting
Hello. I am sorry if this is a common question but through all my searching, I haven't found an answer which matches what I want to do.
I am looking for a sed command that will parse through a large text file and extract lines that start with specific words (which are repeated throughout the... (4 Replies)
Discussion started by: MrDumbQuestion
4 Replies
7. Shell Programming and Scripting
I can't decide if I should use AWK or PERL after pouring over these forums for hours today I decided I'd post something and see if I couldn't get some advice.
I've got a text file full of hundreds of events in this format:
Record Number : 1
Records in Seq : ... (3 Replies)
Discussion started by: Mayday22
3 Replies
8. UNIX for Dummies Questions & Answers
Hi,
I am trying to extract lines from a text file given a text file containing line numbers to be extracted from the first file. How do I go about doing this? Thanks! (1 Reply)
Discussion started by: evelibertine
1 Replies
9. Shell Programming and Scripting
Hello,
I have a file ff.txt that looks as follows
*ABNA.txt
356
24
36
112
*AC24.txt
457
458
321
2
ABNA.txt and AC24.txt are the files in the folder named foo1. Based on the numbers in the ff.txt file, I want to extract the lines from the corresponding files in the foo1 folder and... (2 Replies)
Discussion started by: mohamad
2 Replies
10. Shell Programming and Scripting
Hi Folks,
I have the below feed file named abc1.txt in which you can see there is a title and below is the respective values in the rows and it is completely pipe delimited file ,.
... (4 Replies)
Discussion started by: punpun66
4 Replies
LEARN ABOUT SUSE
afp_signature.conf
AFP_SIGNATURE.CONF(5) Netatalk 2.2 AFP_SIGNATURE.CONF(5)
NAME
afp_signature.conf - Configuration file used by afpd(8) to specify server signature
DESCRIPTION
/etc/netatalk/afp_signature.conf is the configuration file used by afpd to specify server signature automagically. The configuration lines
are composed like:
"server name" hexa-string
The first field is server name. Server names must be quoted if they contain spaces. The second field is the hexadecimal string of 32
characters for 16-bytes server signature.
The leading spaces and tabs are ignored. Blank lines are ignored. The lines prefixed with # are ignored. The illegal lines are ignored.
Note
Server Signature is unique 16-bytes identifier used to prevent logging on to the same server twice.
Netatalk 2.0 and earlier generated server signature by using gethostid(). There was a problem that another servers have the same
signature because the hostid is not unique enough.
Current netatalk generates the signature from random numbers and saves it into afp_signature.conf. When starting next time, it is read
from this file.
This file should not be thoughtlessly edited and be copied onto another server. If it wants to set the signature intentionally, use the
option "-signature user:xxxx" in afpd.conf. In this case, afp_signature.conf is not used.
EXAMPLES
Example. afp_signature.conf three servers on one netatalk
# This is a comment.
"Guest Server" 251D25569CB29AFDF296C6B5B3A385B7
"User Server" 26B5D785046C019F55718F76B1C13BFB
special 74A0BB94EC8C13988B2E75042347E528
SEE ALSO
afpd(8), afpd.conf(5), asip-status.pl(1)
Netatalk 2.2 08 Mar 2011 AFP_SIGNATURE.CONF(5)