Sponsored Content
Full Discussion: Avoid Duplicates in a file
Top Forums UNIX for Dummies Questions & Answers Avoid Duplicates in a file Post 302178529 by ShawnMilo on Tuesday 25th of March 2008 07:49:12 PM
Old 03-25-2008
All you need to trim a file down to unique lines is:

Code:
sort -u file

However, your lines aren't straight duplicates, so you'll need to pre-process the file. If your lines will all be the same except for the initial "reader" part, then just strip out any occurrence of READER_X_X_X.

ShawnMilo
 

10 More Discussions You Might Find Interesting

1. Ubuntu

Avoid creating temporary files on editing a file in Ubuntu

Hi, My ubuntu flavor always create temporary files having filename followed by ~ on editing. For eg: if I am editing a file called "sip.c", automatically a temporary (bkup) file is getting created with the name "sip.c~". How to avoid this file creation? (7 Replies)
Discussion started by: royalibrahim
7 Replies

2. Shell Programming and Scripting

How to avoid a temp file

Hi all. I want to check the free space on a given FS and process the output. Right now, I'm using a temp file to avoid using df twice. This is what I'm doing #!/usr/bin/ksh ... df -k $FS_NAME > $TMP_FILE 2>&1 if ]; then RESULT="CRITICAL - $(cat $TMP_FILE)" else cat $TMP_FILE | ...... (3 Replies)
Discussion started by: fox1212
3 Replies

3. HP-UX

Parameter to avoid file being deleted by SAM

Good afternoon. I am a newbie. We just had a potentially big problem (negated to having good backups). Basically, there is an option in SAM, to delete all the data from the system that a user ever created. Lo and behold, silly me, I choose that option, and all sorts of needed files... (5 Replies)
Discussion started by: instant000
5 Replies

4. Shell Programming and Scripting

Avoid file creation in a script...achive same result

Guys following lines help me in getting numbers from PID column ,to be thrown into first column of a CSV file. COLUMNS=2047 /usr/bin/ps -eo pid,ppid,uid,user,args | grep -v "PID" > /tmp/masterPID.txt cat /tmp/masterPID.txt|while read line do PID=`echo $line|awk '{print $1}'` echo "$PID"... (4 Replies)
Discussion started by: ak835
4 Replies

5. Shell Programming and Scripting

avoid open file to check field.

Hi Everyone, # cat a.txt 94,aqqc,62345907, 5,aeec,77, # cat 1.pl #!/usr/bin/perl use strict; use warnings; use Date::Manip; open(my $FA, "/root/a.txt") or die "$!"; while(<$FA>) { chomp; my @tmp=split(/\,/, $_); if (index($tmp, "qq") ne -1) { ... (4 Replies)
Discussion started by: jimmy_y
4 Replies

6. Shell Programming and Scripting

How to avoid Newline character in generated text file?

Hi All, Just need small help in resolving the special new line character in generated output file. In one of my shell script I am using following lines to get the spool file (i.e. sfile.txt) and AAA_XXXX_p111_n222.txt AAA_YYYY_p111_n222.txt Here assuming v_pnum="p111" v_nid="n222" ... (1 Reply)
Discussion started by: shekharjchandra
1 Replies

7. UNIX for Dummies Questions & Answers

CSV file:Find duplicates, save original and duplicate records in a new file

Hi Unix gurus, Maybe it is too much to ask for but please take a moment and help me out. A very humble request to you gurus. I'm new to Unix and I have started learning Unix. I have this project which is way to advanced for me. File format: CSV file File has four columns with no header... (8 Replies)
Discussion started by: arvindosu
8 Replies

8. Programming

[c]Why first file is creating after the second. How to avoid

Hi, My Code is as below: nbECRITS = fwrite(strstr(data->buffer, ";") + 1, sizeof(char), (data->buffsize) - LEN_NOM_FIC, fic_sortie); fclose(fic_sortie); sprintf(PATH_BALISE, "%s.balise", PATH); fic_balise_data = fopen(PATH_BALISE, "a+"); if (fic_balise_data == NULL) {... (1 Reply)
Discussion started by: ezee
1 Replies

9. Shell Programming and Scripting

Avoid overwriting backup file when multiple entries need to replace in one file input from another

Hello, I have been working on script in which search and replace the multiple pattern. 1. update_params.sh read the multiple pattern from input file ParamMapping.txt(old_entry|New_entry) and passing this values one by one to change_text.sh 2. change_text.sh read... (0 Replies)
Discussion started by: ketanraut
0 Replies

10. Shell Programming and Scripting

Write only changes to file - avoid duplicates

I want to create a file, to save a list of fail2ban blocked ip addresses. So I thought I'd create a loop that will check with fail2ban every minute, and write the ip addresses to a file. while true; do echo $(fail2ban-client status asterisk-iptables | grep 'IP list' | sed 's/.*://g' | sed -e... (4 Replies)
Discussion started by: aristosv
4 Replies
DBVIEW(1)							Database Management							 DBVIEW(1)

NAME
dbview - View dBase III files SYNOPSIS
dbview [-b|--browse] [-d delim| --delimiter delim] [-D|--deleted] [-e|--description] [-h|--help] [-i|--info] [-o|--omit] [-v|--version] [-r|--reserve] [-t|--trim] dbfile DESCRIPTION
Dbview is a little tool that will display dBase III files. You can also use it to convert your old .dbf files for further use with Unix. It should also work with dBase IV files, but this is mostly untested. By default dbview displays the contents of a dBase III or IV database file. This is be done by displaying both the name of the field itself and its value. At the end of every record a newline is appended. OPTIONS
If no option given dbview only displays the database in its most friendly way. --browse, -b switches into browse mode. Using this mode no fieldnames will be displayed, instead every record will displayed in one line using a delimiter to separate fields. --delimiter, -d delimiter The default delimiter in browse mode is the colon sign ``:''. This parameter overrides it. This can be useful especially if you plan to examine the output with scripts. --deleted, -D displays deleted records as well as the delete state in each record in the database. --description, -e displays the field description of the database. --help, -h displays a complete (or short) help screen. --info, -i displays some (partially technical) information about the database like number of records and length of each record. --omit, -o omits displaying the whole database. Using this parameter can be useful if you're only interested in the structure. --reserve, -r Normally fieldnames are converted into a more friendly format. They are stored in capital letters, but that looks like shouting. This parameter supresses the conversion. --trim, -t When this option is specified, leading and trailing spaces are omitted. This might be useful when in browse mode. --version, -v displays version and exits. NOTES
As dBase is DOS, umlauts are stored using a different code table (namely ASCII) than most modern unices (namely ANSI). If you encounter such a file, I would recommend piping the output through recode(1) with ibmpc:latin1 as it's argument. If you want to examine the output generated by the browse mode, just take cut(1) and set its delimiter to the used delimiter or take awk(1) and continue. COPYRIGHT
Dbview is free software. It is based on routines from unknown source that I found on nic.funet.fi in /pub/msdos/languages/c as dbase.c. The file contained the following notice: These functions are provided by Valour Software as a gift. I have modified and included this file and wrote a skeleton around it. All together provides a powerful tool for dBase III and IV database manipulation under Unix. I mainly have written this program, because I've got several dbase files containing important information for me. As I won't go running DOS everytime I need some of the stored information, I had to find a viewer that runs unter Unix, resp. Linux, but unfortunately didn't find one. So it was my turn. This package as a whole is published under the GNU Public License, which is a great invention. It wasn't the intention to write a freaking viewer and reinvent the wheel again. Instead dbview is intend to be used in conjunction with your favourite unix text utilities like cut, recode and more. Martin Schulze Infodrom Oldenburg joey@infodrom.north.de SEE ALSO
recode(1), more(1), awk(1), cut(1). Unix November 20th, 2006 DBVIEW(1)
All times are GMT -4. The time now is 11:26 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy