Sponsored Content
Full Discussion: Avoid Duplicates in a file
Top Forums UNIX for Dummies Questions & Answers Avoid Duplicates in a file Post 302178529 by ShawnMilo on Tuesday 25th of March 2008 07:49:12 PM
Old 03-25-2008
All you need to trim a file down to unique lines is:

Code:
sort -u file

However, your lines aren't straight duplicates, so you'll need to pre-process the file. If your lines will all be the same except for the initial "reader" part, then just strip out any occurrence of READER_X_X_X.

ShawnMilo
 

10 More Discussions You Might Find Interesting

1. Ubuntu

Avoid creating temporary files on editing a file in Ubuntu

Hi, My ubuntu flavor always create temporary files having filename followed by ~ on editing. For eg: if I am editing a file called "sip.c", automatically a temporary (bkup) file is getting created with the name "sip.c~". How to avoid this file creation? (7 Replies)
Discussion started by: royalibrahim
7 Replies

2. Shell Programming and Scripting

How to avoid a temp file

Hi all. I want to check the free space on a given FS and process the output. Right now, I'm using a temp file to avoid using df twice. This is what I'm doing #!/usr/bin/ksh ... df -k $FS_NAME > $TMP_FILE 2>&1 if ]; then RESULT="CRITICAL - $(cat $TMP_FILE)" else cat $TMP_FILE | ...... (3 Replies)
Discussion started by: fox1212
3 Replies

3. HP-UX

Parameter to avoid file being deleted by SAM

Good afternoon. I am a newbie. We just had a potentially big problem (negated to having good backups). Basically, there is an option in SAM, to delete all the data from the system that a user ever created. Lo and behold, silly me, I choose that option, and all sorts of needed files... (5 Replies)
Discussion started by: instant000
5 Replies

4. Shell Programming and Scripting

Avoid file creation in a script...achive same result

Guys following lines help me in getting numbers from PID column ,to be thrown into first column of a CSV file. COLUMNS=2047 /usr/bin/ps -eo pid,ppid,uid,user,args | grep -v "PID" > /tmp/masterPID.txt cat /tmp/masterPID.txt|while read line do PID=`echo $line|awk '{print $1}'` echo "$PID"... (4 Replies)
Discussion started by: ak835
4 Replies

5. Shell Programming and Scripting

avoid open file to check field.

Hi Everyone, # cat a.txt 94,aqqc,62345907, 5,aeec,77, # cat 1.pl #!/usr/bin/perl use strict; use warnings; use Date::Manip; open(my $FA, "/root/a.txt") or die "$!"; while(<$FA>) { chomp; my @tmp=split(/\,/, $_); if (index($tmp, "qq") ne -1) { ... (4 Replies)
Discussion started by: jimmy_y
4 Replies

6. Shell Programming and Scripting

How to avoid Newline character in generated text file?

Hi All, Just need small help in resolving the special new line character in generated output file. In one of my shell script I am using following lines to get the spool file (i.e. sfile.txt) and AAA_XXXX_p111_n222.txt AAA_YYYY_p111_n222.txt Here assuming v_pnum="p111" v_nid="n222" ... (1 Reply)
Discussion started by: shekharjchandra
1 Replies

7. UNIX for Dummies Questions & Answers

CSV file:Find duplicates, save original and duplicate records in a new file

Hi Unix gurus, Maybe it is too much to ask for but please take a moment and help me out. A very humble request to you gurus. I'm new to Unix and I have started learning Unix. I have this project which is way to advanced for me. File format: CSV file File has four columns with no header... (8 Replies)
Discussion started by: arvindosu
8 Replies

8. Programming

[c]Why first file is creating after the second. How to avoid

Hi, My Code is as below: nbECRITS = fwrite(strstr(data->buffer, ";") + 1, sizeof(char), (data->buffsize) - LEN_NOM_FIC, fic_sortie); fclose(fic_sortie); sprintf(PATH_BALISE, "%s.balise", PATH); fic_balise_data = fopen(PATH_BALISE, "a+"); if (fic_balise_data == NULL) {... (1 Reply)
Discussion started by: ezee
1 Replies

9. Shell Programming and Scripting

Avoid overwriting backup file when multiple entries need to replace in one file input from another

Hello, I have been working on script in which search and replace the multiple pattern. 1. update_params.sh read the multiple pattern from input file ParamMapping.txt(old_entry|New_entry) and passing this values one by one to change_text.sh 2. change_text.sh read... (0 Replies)
Discussion started by: ketanraut
0 Replies

10. Shell Programming and Scripting

Write only changes to file - avoid duplicates

I want to create a file, to save a list of fail2ban blocked ip addresses. So I thought I'd create a loop that will check with fail2ban every minute, and write the ip addresses to a file. while true; do echo $(fail2ban-client status asterisk-iptables | grep 'IP list' | sed 's/.*://g' | sed -e... (4 Replies)
Discussion started by: aristosv
4 Replies
UNIFDEF(1)						      General Commands Manual							UNIFDEF(1)

NAME
unifdef - remove ifdef'ed lines SYNOPSIS
unifdef [ -t -l -c -Dsym -Usym -idsym -iusym ] ... [ file ] DESCRIPTION
Unifdef is useful for removing ifdef'ed lines from a file while otherwise leaving the file alone. Unifdef is like a stripped-down C pre- processor: it is smart enough to deal with the nested ifdefs, comments, single and double quotes of C syntax so that it can do its job, but it doesn't do any including or interpretation of macros. Neither does it strip out comments, though it recognizes and ignores them. You specify which symbols you want defined -Dsym or undefined -Usym and the lines inside those ifdefs will be copied to the output or removed as appropriate. The ifdef, ifndef, else, and endif lines associated with sym will also be removed. Ifdefs involving symbols you don't specify are untouched and copied out along with their associated ifdef, else, and endif lines. If an ifdef X occurs nested inside another ifdef X, then the inside ifdef is treated as if it were an unrecognized symbol. If the same symbol appears in more than one argument, only the first occurrence is significant. The -l option causes unifdef to replace removed lines with blank lines instead of deleting them. If you use ifdefs to delimit non-C lines, such as comments or code which is under construction, then you must tell unifdef which symbols are used for that purpose so that it won't try to parse for quotes and comments in those ifdef'ed lines. You specify that you want the lines inside certain ifdefs to be ignored but copied out with -idsym and -iusym similar to -Dsym and -Usym above. If you want to use unifdef for plain text (not C code), use the -t option. This makes unifdef refrain from attempting to recognize com- ments and single and double quotes. Unifdef copies its output to stdout and will take its input from stdin if no file argument is given. If the -c argument is specified, then the operation of unifdef is complemented, i.e. the lines that would have been removed or blanked are retained and vice versa. SEE ALSO
diff(1) DIAGNOSTICS
Premature EOF, inappropriate else or endif. Exit status is 0 if output is exact copy of input, 1 if not, 2 if trouble. BUGS
Does not know how to deal with cpp consructs such as #if defined(X) || defined(Y) AUTHOR
Dave Yost 4.3 Berkeley Distribution April 29, 1985 UNIFDEF(1)
All times are GMT -4. The time now is 08:39 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy