Sponsored Content
Full Discussion: Avoid Duplicates in a file
Top Forums UNIX for Dummies Questions & Answers Avoid Duplicates in a file Post 302178031 by sysgate on Monday 24th of March 2008 10:27:42 AM
Old 03-24-2008
use :
Code:
sort file1 | uniq -u > new-file-only-unique-lines

"uniq -q" prints only the unique lines in a file.
 

10 More Discussions You Might Find Interesting

1. Ubuntu

Avoid creating temporary files on editing a file in Ubuntu

Hi, My ubuntu flavor always create temporary files having filename followed by ~ on editing. For eg: if I am editing a file called "sip.c", automatically a temporary (bkup) file is getting created with the name "sip.c~". How to avoid this file creation? (7 Replies)
Discussion started by: royalibrahim
7 Replies

2. Shell Programming and Scripting

How to avoid a temp file

Hi all. I want to check the free space on a given FS and process the output. Right now, I'm using a temp file to avoid using df twice. This is what I'm doing #!/usr/bin/ksh ... df -k $FS_NAME > $TMP_FILE 2>&1 if ]; then RESULT="CRITICAL - $(cat $TMP_FILE)" else cat $TMP_FILE | ...... (3 Replies)
Discussion started by: fox1212
3 Replies

3. HP-UX

Parameter to avoid file being deleted by SAM

Good afternoon. I am a newbie. We just had a potentially big problem (negated to having good backups). Basically, there is an option in SAM, to delete all the data from the system that a user ever created. Lo and behold, silly me, I choose that option, and all sorts of needed files... (5 Replies)
Discussion started by: instant000
5 Replies

4. Shell Programming and Scripting

Avoid file creation in a script...achive same result

Guys following lines help me in getting numbers from PID column ,to be thrown into first column of a CSV file. COLUMNS=2047 /usr/bin/ps -eo pid,ppid,uid,user,args | grep -v "PID" > /tmp/masterPID.txt cat /tmp/masterPID.txt|while read line do PID=`echo $line|awk '{print $1}'` echo "$PID"... (4 Replies)
Discussion started by: ak835
4 Replies

5. Shell Programming and Scripting

avoid open file to check field.

Hi Everyone, # cat a.txt 94,aqqc,62345907, 5,aeec,77, # cat 1.pl #!/usr/bin/perl use strict; use warnings; use Date::Manip; open(my $FA, "/root/a.txt") or die "$!"; while(<$FA>) { chomp; my @tmp=split(/\,/, $_); if (index($tmp, "qq") ne -1) { ... (4 Replies)
Discussion started by: jimmy_y
4 Replies

6. Shell Programming and Scripting

How to avoid Newline character in generated text file?

Hi All, Just need small help in resolving the special new line character in generated output file. In one of my shell script I am using following lines to get the spool file (i.e. sfile.txt) and AAA_XXXX_p111_n222.txt AAA_YYYY_p111_n222.txt Here assuming v_pnum="p111" v_nid="n222" ... (1 Reply)
Discussion started by: shekharjchandra
1 Replies

7. UNIX for Dummies Questions & Answers

CSV file:Find duplicates, save original and duplicate records in a new file

Hi Unix gurus, Maybe it is too much to ask for but please take a moment and help me out. A very humble request to you gurus. I'm new to Unix and I have started learning Unix. I have this project which is way to advanced for me. File format: CSV file File has four columns with no header... (8 Replies)
Discussion started by: arvindosu
8 Replies

8. Programming

[c]Why first file is creating after the second. How to avoid

Hi, My Code is as below: nbECRITS = fwrite(strstr(data->buffer, ";") + 1, sizeof(char), (data->buffsize) - LEN_NOM_FIC, fic_sortie); fclose(fic_sortie); sprintf(PATH_BALISE, "%s.balise", PATH); fic_balise_data = fopen(PATH_BALISE, "a+"); if (fic_balise_data == NULL) {... (1 Reply)
Discussion started by: ezee
1 Replies

9. Shell Programming and Scripting

Avoid overwriting backup file when multiple entries need to replace in one file input from another

Hello, I have been working on script in which search and replace the multiple pattern. 1. update_params.sh read the multiple pattern from input file ParamMapping.txt(old_entry|New_entry) and passing this values one by one to change_text.sh 2. change_text.sh read... (0 Replies)
Discussion started by: ketanraut
0 Replies

10. Shell Programming and Scripting

Write only changes to file - avoid duplicates

I want to create a file, to save a list of fail2ban blocked ip addresses. So I thought I'd create a loop that will check with fail2ban every minute, and write the ip addresses to a file. while true; do echo $(fail2ban-client status asterisk-iptables | grep 'IP list' | sed 's/.*://g' | sed -e... (4 Replies)
Discussion started by: aristosv
4 Replies
comm(1) 							   User Commands							   comm(1)

NAME
comm - select or reject lines common to two files SYNOPSIS
comm [-123] file1 file2 DESCRIPTION
The comm utility reads file1 and file2, which must be ordered in the current collating sequence, and produces three text columns as output: lines only in file1; lines only in file2; and lines in both files. If the input files were ordered according to the collating sequence of the current locale, the lines written will be in the collating sequence of the original lines. If not, the results are unspecified. OPTIONS
The following options are supported: -1 Suppresses the output column of lines unique to file1. -2 Suppresses the output column of lines unique to file2. -3 Suppresses the output column of lines duplicated in file1 and file2. OPERANDS
The following operands are supported: file1 A path name of the first file to be compared. If file1 is -, the standard input is used. file2 A path name of the second file to be compared. If file2 is -, the standard input is used. USAGE
See largefile(5) for the description of the behavior of comm when encountering files greater than or equal to 2 Gbyte ( 2**31 bytes). EXAMPLES
Example 1: Printing a list of utilities specified by files If file1, file2, and file3 each contain a sorted list of utilities, the command example% comm -23 file1 file2 | comm -23 - file3 prints a list of utilities in file1 not specified by either of the other files. The entry: example% comm -12 file1 file2 | comm -12 - file3 prints a list of utilities specified by all three files. And the entry: example% comm -12 file2 file3 | comm -23 -file1 prints a list of utilities specified by both file2 and file3, but not specified in file1. ENVIRONMENT VARIABLES
See environ(5) for descriptions of the following environment variables that affect the execution of comm: LANG, LC_ALL, LC_COLLATE, LC_CTYPE, LC_MESSAGES, and NLSPATH. EXIT STATUS
The following exit values are returned: 0 All input files were successfully output as specified. >0 An error occurred. ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWesu | +-----------------------------+-----------------------------+ |CSI |enabled | +-----------------------------+-----------------------------+ |Interface Stability |Standard | +-----------------------------+-----------------------------+ SEE ALSO
cmp(1), diff(1), sort(1), uniq(1), attributes(5), environ(5), largefile(5), standards(5) SunOS 5.10 3 Mar 2004 comm(1)
All times are GMT -4. The time now is 06:48 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy