Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Using an awk script to identify dupes in two files
Top Forums Shell Programming and Scripting Using an awk script to identify dupes in two files Post 302499027 by gimley on Wednesday 23rd of February 2011 05:18:55 AM
Old 02-23-2011
Hello,
Many thanks for the solution and also the prompt reply.
The script worked beautifully with the sample data. But with real world data it did not work. Should I have mentioned that the right hand side of the master file is in upper ascii or possibly in Unicode code pages other than Latin 1
Code:
aabad=¥ÊÚÄ
aabadaabu=¥ÊÚÄÚÊÞ
aabadabu=¥ÊÚÄÚÊÞ
aabbey=¥ÊèÊá
aabedali=¥ÊáĤÑÜ
aabedin=¥ÊáÄÛÆ
aabeed=¥ÊÜÄ
aabel=¥ÊáÑ
aabelia=¥ÊáÑÛÍÚ
aabelin=¥ÊáÑÜÆ

and the "slave" file has only lower ascii but no upper ascii
Code:
kishor
aabaadabu
aabhar

The output should have been
Code:
kishor
aabhar

Any solutions please,

Best regards and sorry for the hassle,

Gimley
Last edited by Franklin52; 02-24-2011 at 03:22 AM.. Reason: Please use code tags
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

how to identify files I cannot access

I want to use find (or something else) to give me a list of all files in a directory tree where the group access is not rwx or rw-. I'm trying to archive the whole directory tree, but it won't archive any files where I do not have at least read access. I have tried: find . ! -perm -060 but... (4 Replies)
Discussion started by: wvdeijk
4 Replies

2. Shell Programming and Scripting

Need help to identify files

Hi, I have one problem. I want to identify all those files which are named according to the format <name>_<date>. I have tried using awk and grep in bash but i m not able to get it correct. Can someone please help? It's urgent !! (10 Replies)
Discussion started by: udayan_goswami
10 Replies

3. Shell Programming and Scripting

Shell script to identify the number of files and to append data

Hi I am having a question where I have to 1) Identify the number of files in a directory with a specific format and if the count is >1 we need to concatenate those two files into one file and remember that in the second file the header should not be copied. it should be form first file.... (4 Replies)
Discussion started by: pradkumar
4 Replies

4. Shell Programming and Scripting

Perl Script to identify files without extension and assign a value

Hi, I have a perl script which is a part of a shell script which read lines from a flat file(which is generated as part of a script after a series of bteq/fexp) and assigns a value for each object in the file based on the type of file name. (i.e extensions like .bteq/.ctl/.ksh etc) For example,... (1 Reply)
Discussion started by: yohasini
1 Replies

5. Shell Programming and Scripting

Script for identifying and deleting dupes in a line

I am compiling a synonym dictionary which has the following structure Headword=Synonym1,Synonym2 and so on, with each synonym separated by a comma. As is usual in such cases manual preparation of synonyms results in repeating the synonym which results in dupes as in the example below:... (3 Replies)
Discussion started by: gimley
3 Replies

6. Shell Programming and Scripting

Removing Dupes from huge file- awk/perl/uniq

Hi, I have the following command in place nawk -F, '!a++' file > file.uniq It has been working perfectly as per requirements, by removing duplicates by taking into consideration only first 3 fields. Recently it has started giving below error: bash-3.2$ nawk -F, '!a++'... (17 Replies)
Discussion started by: makn
17 Replies

7. Shell Programming and Scripting

Help in modifying existing Perl Script to produce report of dupes

Hello, I have a large amount of data with the following structure: Word=Transliterated word I have written a Perl Script (reproduced below) which goes through the full file and identifies all dupes on the right hand side. It creates successfully a new file with two headers: Singletons and Dupes.... (5 Replies)
Discussion started by: gimley
5 Replies

8. Shell Programming and Scripting

Bash script to find the number of files and identify which ones are 0 bytes.

I am writing a bash script to find out all the files in a directory which are empty. I am running into multiple issues. I will really appreciate if someone can please help me. #!/bin/bash DATE=$(date +%m%d%y) TIME=$(date +%H%M) DIR="/home/statsetl/input/civil/test" ... (1 Reply)
Discussion started by: monasharma13
1 Replies

9. Shell Programming and Scripting

Help with Perl script for identifying dupes in column1

Dear all, I have a large dictionary database which has the following structure source word=target word e.g. book=livre Since the database is very large in spite of all the care taken, it so happens that at times the source word is repeated e.g. book=livre book=tome Since I want to... (7 Replies)
Discussion started by: gimley
7 Replies

10. Shell Programming and Scripting

Modify script to remove dupes with two delimiters

Hello, I have a script which removes duplicates in a database with a single delimiter = The script is given below: # script to remove dupes from a row with structure word=word BEGIN{FS="="} {for(i=1;i<=NF;i++){a++;}for(i in a){b=b"="i}{sub("=","",b);$0=b;b="";delete a}}1 How do I modify... (6 Replies)
Discussion started by: gimley
6 Replies

LEARN ABOUT CENTOS

locale::codes::langfam

Locale::Codes::LangFam(3)				User Contributed Perl Documentation				 Locale::Codes::LangFam(3)

NAME

       Locale::Codes::LangFam - standard codes for language extension identification

SYNOPSIS

	  use Locale::Codes::LangFam;

	  $lext = code2langfam('apa');		       # $lext gets 'Apache languages'
	  $code = langfam2code('Apache languages');    # $code gets 'apa'

	  @codes   = all_langfam_codes();
	  @names   = all_langfam_names();

DESCRIPTION

       The "Locale::Codes::LangFam" module provides access to standard codes used for identifying language families, such as those as defined in
       ISO 639-5.

       Most of the routines take an optional additional argument which specifies the code set to use. If not specified, the default ISO 639-5
       language family codes will be used.

SUPPORTED CODE SETS

       There are several different code sets you can use for identifying language families. A code set may be specified using either a name, or a
       constant that is automatically exported by this module.

       For example, the two are equivalent:

	  $lext = code2langfam('apa','alpha');
	  $lext = code2langfam('apa',LOCALE_LANGFAM_ALPHA);

       The codesets currently supported are:

       alpha
	   This is the set of three-letter (lowercase) codes from ISO 639-5 such as 'apa' for Apache languages.

	   This is the default code set.

ROUTINES

       code2langfam ( CODE [,CODESET] )
       langfam2code ( NAME [,CODESET] )
       langfam_code2code ( CODE ,CODESET ,CODESET2 )
       all_langfam_codes ( [CODESET] )
       all_langfam_names ( [CODESET] )
       Locale::Codes::LangFam::rename_langfam  ( CODE ,NEW_NAME [,CODESET] )
       Locale::Codes::LangFam::add_langfam  ( CODE ,NAME [,CODESET] )
       Locale::Codes::LangFam::delete_langfam  ( CODE [,CODESET] )
       Locale::Codes::LangFam::add_langfam_alias  ( NAME ,NEW_NAME )
       Locale::Codes::LangFam::delete_langfam_alias  ( NAME )
       Locale::Codes::LangFam::rename_langfam_code  ( CODE ,NEW_CODE [,CODESET] )
       Locale::Codes::LangFam::add_langfam_code_alias  ( CODE ,NEW_CODE [,CODESET] )
       Locale::Codes::LangFam::delete_langfam_code_alias  ( CODE [,CODESET] )
	   These routines are all documented in the Locale::Codes::API man page.

SEE ALSO

       Locale::Codes
	   The Locale-Codes distribution.

       Locale::Codes::API
	   The list of functions supported by this module.

       http://www.loc.gov/standards/iso639-5/id.php
	   ISO 639-5 .

AUTHOR

       See Locale::Codes for full author history.

       Currently maintained by Sullivan Beck (sbeck@cpan.org).

COPYRIGHT

	  Copyright (c) 2011-2013 Sullivan Beck

       This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

perl v5.16.3							    2013-02-27						 Locale::Codes::LangFam(3)
All times are GMT -4. The time now is 12:40 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy