Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: UNIX scripting for finding duplicates and null records in pk columns
Top Forums Shell Programming and Scripting UNIX scripting for finding duplicates and null records in pk columns Post 302901036 by Yoda on Sunday 11th of May 2014 01:48:11 AM
Old 05-11-2014
Here is an awk approach:
Code:
awk -F\| '
        NR == 1 {
                next
        }
        {
                I = $1 OFS $2
                if ( ( I in U ) || !($1 && $2) )
                        print $0 > "dupl.txt"
        }
        $1 && $2 {
                U[I] = $0
        }
        END {
                for ( k in U )
                        print U[k] > "uniq.txt"
        }
' abc.txt

This program creates output files: dupl.txt and uniq.txt with duplicate and unique records.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

finding duplicates with perl

I have a huge file (over 30mb) that I am processing through with perl. I am pulling out a list of filenames and placing it in an array called @reports. I am fine up till here. What I then want to do is go through the array and find any duplicates. If there is a duplicate, output it to the screen.... (3 Replies)
Discussion started by: dangral
3 Replies

2. Shell Programming and Scripting

finding null records in data file

I am having a "|" delimited flat file and I have to pick up all the records with the 2nd field having null value. Please suggest. (3 Replies)
Discussion started by: dsravan
3 Replies

3. Shell Programming and Scripting

finding duplicates in columns and removing lines

I am trying to figure out how to scan a file like so: 1 ralphs office","555-555-5555","ralph@mail.com","www.ralph.com 2 margies office","555-555-5555","ralph@mail.com","www.ralph.com 3 kims office","555-555-5555","kims@mail.com","www.ralph.com 4 tims... (17 Replies)
Discussion started by: totus
17 Replies

4. Shell Programming and Scripting

Finding duplicates from positioned substring across lines

I have million's of records each containing exactly 50 characters and have to check the uniqueness of 4 character substring of 50 character (postion known prior) and report if any duplicates are found. Eg. data... AAAA00000000000000XXXX0000 0000000000... upto50 chars... (2 Replies)
Discussion started by: gapprasath
2 Replies

5. Shell Programming and Scripting

exclude records with null fields

Hi, can I do something like this to add a condition of checking if the 4th field is number or space or blank also: awk -F, '$4 /^*||*/' MYFILE >> OTHERFILE I also want the other part i.e. I need to exclude all lines whose 4th field is space or blank or number: MYFILE a,b,c,d,e a,b,c,2,r... (2 Replies)
Discussion started by: praveenK_Dudala
2 Replies

6. Shell Programming and Scripting

Awk to Count Records with not null

Hi, I have a pipe seperated file I want to write a code to display count of lines that have 20th field not null. nawk -F"|" '{if ($20!="") print NR,$20}' xyz..txt This displays records with 20th field also null. I would like output as: (4 Replies)
Discussion started by: pinnacle
4 Replies

7. Shell Programming and Scripting

Unix sort for fixed length columns and records

I was trying to use the AIX 6.1 sort command to sort fixed-length data records, sorting by specific columns only. It took some time to figure out how to get it to work, so I wanted to share the solution. The sort man page wasn't much help, because it talks about field delimeters (default space... (1 Reply)
Discussion started by: CheeseHead1
1 Replies

8. Shell Programming and Scripting

Help finding non duplicates

I am currently creating a script to find filenames that are listed once in an input file (find non duplicates). I then want to report those single files in another file. Here is the function that I have so far: function dups_filenames { file2="" file1="" file="" dn="" ch="" pn="" ... (6 Replies)
Discussion started by: chipblah84
6 Replies

9. Shell Programming and Scripting

finding duplicates in csv based on key columns

Hi team, I have 20 columns csv files. i want to find the duplicates in that file based on the column1 column10 column4 column6 coulnn8 coulunm2 . if those columns have same values . then it should be a duplicate record. can one help me on finding the duplicates, Thanks in advance. ... (2 Replies)
Discussion started by: baskivs
2 Replies

10. UNIX for Dummies Questions & Answers

Finding duplicates then copying, almost there, maybe?

Hi everyone. I'm trying to help my wife with a project, she has exported 200 images from many different folders, unfortunately there was a problem with the export and I need to find the master versions so that she doesn't have to go through and select them again. I need to: For each image in... (2 Replies)
Discussion started by: Rhinoskin
2 Replies

LEARN ABOUT REDHAT

unix2dos

unix2dos(1)						      General Commands Manual						       unix2dos(1)

NAME

       unix2dos - UNIX to DOS text file format converter

SYNOPSYS

       unix2dos [options] [-c convmode] [-o file ...] [-n infile outfile ...]

       Options:

       [-hkqV] [--help] [--keepdate] [--quiet] [--version]

DESCRIPTION

       This manual page documents unix2dos, the program that converts text files in UNIX format to DOS format.

OPTIONS

       The following options are available:

       -h --help
	      Print online help.

       -k --keepdate
	      Keep the date stamp of output file same as input file.

       -q --quiet
	      Quiet mode. Suppress all warning and messages.

       -V --version
	      Prints version information.

       -c --convmode convmode
	      Sets conversion mode. Simulates unix2dos under SunOS.

       -o --oldfile file ...
	      Old file mode. Convert the file and write output to it. The program default to run in this mode. Wildcard names may be used.

       -n --newfile infile outfile ...
	      New  file  mode.	Convert the infile and write output to outfile. File names must be given in pairs and wildcard names should NOT be
	      used or you WILL lost your files.

EXAMPLES

       Get input from stdin and write output to stdout.

	      unix2dos

       Convert and replace a.txt. Convert and replace b.txt.

	      unix2dos a.txt b.txt

	      unix2dos -o a.txt b.txt

       Convert and replace a.txt in ASCII conversion mode.  Convert and replace b.txt in ISO conversion mode.

	      unix2dos a.txt -c iso b.txt

	      unix2dos -c ascii a.txt -c iso b.txt

       Convert and replace a.txt while keeping original date stamp.

	      unix2dos -k a.txt

	      unix2dos -k -o a.txt

       Convert a.txt and write to e.txt.

	      unix2dos -n a.txt e.txt

       Convert a.txt and write to e.txt, keep date stamp of e.txt same as a.txt.

	      unix2dos -k -n a.txt e.txt

       Convert and replace a.txt. Convert b.txt and write to e.txt.

	      unix2dos a.txt -n b.txt e.txt

	      unix2dos -o a.txt -n b.txt e.txt

       Convert c.txt and write to e.txt. Convert and replace a.txt.  Convert and replace b.txt. Convert d.txt and write to f.txt.

	      unix2dos -n c.txt e.txt -o a.txt b.txt -n d.txt f.txt

DIAGNOSTICS

BUGS

       The program does not work properly under MSDOS in stdio processing mode.  If you know why is that so, please tell me.

AUTHOR

       Benjamin Lin - ( blin@socs.uts.edu.au )

MISCELLANY

       Tested environment:

	      Linux 1.2.0 with GNU C 2.5.8

	      SunOS 4.1.3 with GNU C 2.6.3

	      MS-DOS 6.20 with Borland C++ 4.02

       Suggestions and bug reports are welcome.

SEE ALSO

       dos2unix(1)

1995.03.31							   unix2dos v2.2						       unix2dos(1)
All times are GMT -4. The time now is 08:46 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy