Sponsored Content
Top Forums Shell Programming and Scripting extracting unique lines from text file Post 302227942 by soliberus on Friday 22nd of August 2008 09:46:23 AM
Old 08-22-2008
extracting unique lines from text file

I have a file with 14million lines and I would like to extract all the unique lines from the file into another text file.

For example:

Contents of file1

happy
sad
smile
happy
funny
sad

I want to run a command against file one that only returns the unique lines (ie 1 line for happy and 1 line for sad).

Could someone please point me in the right direction.

Thanks
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extracting records with unique fields from a fixed width txt file

Greetings, I would like to extract records from a fixed width text file that have unique field elements. Data is structured like this: John A Smith NY Mary C Jones WA Adam J Clark PA Mary Jones WA Fieldname / start-end position Firstname 1-10... (8 Replies)
Discussion started by: sitney
8 Replies

2. Shell Programming and Scripting

Extracting Text Between Two Unique Lines

Hi all! Im trying to extract a portion of text from a file and put it into a new file. I need all the lines between <Placement> and </Placement> including the Placemark lines themselves. Is there a way to extract all instances of these and not just the first one found? I've tried using sed and... (4 Replies)
Discussion started by: Grizzly
4 Replies

3. Shell Programming and Scripting

awk : extracting unique lines based on columns

Hi, snp.txt CHR_A SNP_A BP_A_st BP_A_End CHR_B BP_B SNP_B R2 p-SNP_A p-SNP_B 5 rs1988728 74904317 74904318 5 74960646 rs1427924 0.377333 0.000740085 0.013930081 5 ... (12 Replies)
Discussion started by: genehunter
12 Replies

4. Shell Programming and Scripting

Extracting several lines of text after a unique string

I'm attempting to write a script to identify users who have sudo access on a server. I only want to extract the ID's of the sudo users after a unique line of text. The list of sudo users goes to the EOF so I only need the script to start after the unique line of text. I already have a script to... (1 Reply)
Discussion started by: bouncer
1 Replies

5. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies

6. Shell Programming and Scripting

Extracting Multiple Lines from a Text File

Hello. I am sorry if this is a common question but through all my searching, I haven't found an answer which matches what I want to do. I am looking for a sed command that will parse through a large text file and extract lines that start with specific words (which are repeated throughout the... (4 Replies)
Discussion started by: MrDumbQuestion
4 Replies

7. Shell Programming and Scripting

Combine multiple unique lines from event log text file into one line, use PERL or AWK?

I can't decide if I should use AWK or PERL after pouring over these forums for hours today I decided I'd post something and see if I couldn't get some advice. I've got a text file full of hundreds of events in this format: Record Number : 1 Records in Seq : ... (3 Replies)
Discussion started by: Mayday22
3 Replies

8. UNIX for Dummies Questions & Answers

Extracting lines from a text file based on another text file with line numbers

Hi, I am trying to extract lines from a text file given a text file containing line numbers to be extracted from the first file. How do I go about doing this? Thanks! (1 Reply)
Discussion started by: evelibertine
1 Replies

9. Shell Programming and Scripting

Extracting lines from text files in folder based on the numbers in another file

Hello, I have a file ff.txt that looks as follows *ABNA.txt 356 24 36 112 *AC24.txt 457 458 321 2 ABNA.txt and AC24.txt are the files in the folder named foo1. Based on the numbers in the ff.txt file, I want to extract the lines from the corresponding files in the foo1 folder and... (2 Replies)
Discussion started by: mohamad
2 Replies

10. Shell Programming and Scripting

Extracting unique values of a column from a feed file

Hi Folks, I have the below feed file named abc1.txt in which you can see there is a title and below is the respective values in the rows and it is completely pipe delimited file ,. ... (4 Replies)
Discussion started by: punpun66
4 Replies
AFP_SIGNATURE.CONF(5)						   Netatalk 2.2 					     AFP_SIGNATURE.CONF(5)

NAME
afp_signature.conf - Configuration file used by afpd(8) to specify server signature DESCRIPTION
/etc/netatalk/afp_signature.conf is the configuration file used by afpd to specify server signature automagically. The configuration lines are composed like: "server name" hexa-string The first field is server name. Server names must be quoted if they contain spaces. The second field is the hexadecimal string of 32 characters for 16-bytes server signature. The leading spaces and tabs are ignored. Blank lines are ignored. The lines prefixed with # are ignored. The illegal lines are ignored. Note Server Signature is unique 16-bytes identifier used to prevent logging on to the same server twice. Netatalk 2.0 and earlier generated server signature by using gethostid(). There was a problem that another servers have the same signature because the hostid is not unique enough. Current netatalk generates the signature from random numbers and saves it into afp_signature.conf. When starting next time, it is read from this file. This file should not be thoughtlessly edited and be copied onto another server. If it wants to set the signature intentionally, use the option "-signature user:xxxx" in afpd.conf. In this case, afp_signature.conf is not used. EXAMPLES
Example. afp_signature.conf three servers on one netatalk # This is a comment. "Guest Server" 251D25569CB29AFDF296C6B5B3A385B7 "User Server" 26B5D785046C019F55718F76B1C13BFB special 74A0BB94EC8C13988B2E75042347E528 SEE ALSO
afpd(8), afpd.conf(5), asip-status.pl(1) Netatalk 2.2 08 Mar 2011 AFP_SIGNATURE.CONF(5)
All times are GMT -4. The time now is 11:21 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy