Sponsored Content
Top Forums Shell Programming and Scripting Remove rows with first 4 fields duplicated in awk Post 302568782 by ahamed101 on Friday 28th of October 2011 08:10:46 AM
Old 10-28-2011
Just extending Jayan's code... Requirement was first 4 fields not 4th field... Smilie
Code:
awk '!x[$1$2$3$4]++'  input_file

--ahamed
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk script to remove duplicate rows in line

i have the long file more than one ns and www and mx in the line like . i need the first ns record and first www and first mx from line . the records are seperated with tthe ; i am try ing in awk scripting not getiing the solution. ... (4 Replies)
Discussion started by: kiranmosarla
4 Replies

2. Shell Programming and Scripting

Help with remove duplicated content

Input file: hcmv-US25-2-3p hsa-3160-5 hcmv-US33 hsa-47 hcmv-UL70-3p hsa-4508 hcmv-UL70-3p hsa-4486 hcms-US25 hsa-360-5 hcms-US25 hsa-4 hcms-US25 hsa-458 hcms-US25 hsa-44812 . . Desired Output file: hcmv-US25-2-3p hsa-3160-5 hcmv-US33 hsa-47 hcmv-UL70-3p hsa-4508 hsa-4486... (3 Replies)
Discussion started by: perl_beginner
3 Replies

3. Shell Programming and Scripting

awk to grep rows by multiple fields

Hello, I met a challenge to extract part of the table. I'd like to grep the first three matches based on field1 and field2. Input: D A 92.85 1315 83 11 D A 95.90 757 28 3 D A 94.38 480 20 7 D A 91.21 307 21 6 D A 94.26 244 ... (6 Replies)
Discussion started by: yifangt
6 Replies

4. Shell Programming and Scripting

How to remove duplicated lines?

Hi, if i have a file like this: Query=1 a a b c c c d Query=2 b b b c c e . . . (7 Replies)
Discussion started by: the_simpsons
7 Replies

5. Shell Programming and Scripting

Delete duplicated fields in a line

Hi, I have files with this kind of format (separator is space): A1 B1 C1 D1 E1 F1 D1 C1 G1 H1 A2 B2 C2 D2 E2 F2 D2 C2 G2 H2 A3 B3 C3 D3 E3 F3 G3 D3 C3 H3 A4 B4 C4 D4 E4 F4 G4 D4 C4 H4 I want the output to be: A1 B1 E1 F1 G1 H1 A2 B2 E2 F2 G2 H2 A3 B3 E3 F3 G3 H3 A4 B4 E4 F4 G4... (12 Replies)
Discussion started by: Gr4wk
12 Replies

6. Shell Programming and Scripting

Removing duplicated first field rows

Hello, I am trying to eliminate rows where the first field is duplicated, leaving the row where the last field is "NET". Data file: 345234|22.34|LST 546543|55.33|LST 793929|98.23|LST 793929|64.69|NET 149593|49.22|LST Desired output: 345234|22.34|LST 546543|55.33|LST... (2 Replies)
Discussion started by: palex
2 Replies

7. Shell Programming and Scripting

Merge files and remove duplicated rows

In a folder I'll several times daily receive new files that I want to combine into one big file, without any duplicate rows. The file name in the folder will look like e.q: MissingData_2014-08-25_09-30-18.txt MissingData_2014-08-25_09-30-14.txt MissingData_2014-08-26_09-30-12.txt The content... (9 Replies)
Discussion started by: Bergans
9 Replies

8. Shell Programming and Scripting

Remove rows containing commas with awk

Hello everyone, I have a dataset that looks something like: 1 3 2 2 3 4,5 4 3:9 5 5,9 6 5:6 I need to remove the rows that contain a comma in the second column and I'm not sure how to go about this. Here is an attempt. awk 'BEGIN {FS=" "} { if ($2!==,) print }'Any help is appreciated. (5 Replies)
Discussion started by: Rabu
5 Replies

9. Shell Programming and Scripting

awk to remove range of fields

I am trying to cut a range of fields in awk. The below seems to work for removing field 50, but what is the correct syntax for removing a range ($50-$62). Thank you :). awk awk 'BEGIN{FS=OFS="\t"}{$50=""; gsub(/\t\t/,"\t")}1' test.vcf.hg19_multianno.txt > output.csv Maybe: awk... (6 Replies)
Discussion started by: cmccabe
6 Replies

10. Shell Programming and Scripting

awk to remove lines where field count is greather than 1 in two fields

I am trying to remove all the lines and spaces where the count in $4 or $5 is greater than 1 (more than 1 letter). The file and the output are tab-delimited. Thank you :). file X 5811530 . G C NLGN4X 17 10544696 . GA G MYH3 9 96439004 . C ... (1 Reply)
Discussion started by: cmccabe
1 Replies
DBIx::Class::Optional::Dependencies(3)			User Contributed Perl Documentation		    DBIx::Class::Optional::Dependencies(3)

NAME
DBIx::Class::Optional::Dependencies - Optional module dependency specifications (for module authors) SYNOPSIS
Somewhere in your build-file (e.g. Module::Install's Makefile.PL): ... configure_requires 'DBIx::Class' => '0.08270'; require DBIx::Class::Optional::Dependencies; my $deploy_deps = DBIx::Class::Optional::Dependencies->req_list_for('deploy'); for (keys %$deploy_deps) { requires $_ => $deploy_deps->{$_}; } ... Note that there are some caveats regarding "configure_requires()", more info can be found at "configure_requires" in Module::Install DESCRIPTION
Some of the less-frequently used features of DBIx::Class have external module dependencies on their own. In order not to burden the average user with modules he will never use, these optional dependencies are not included in the base Makefile.PL. Instead an exception with a descriptive message is thrown when a specific feature is missing one or several modules required for its operation. This module is the central holding place for the current list of such dependencies, for DBIx::Class core authors, and DBIx::Class extension authors alike. CURRENT REQUIREMENT GROUPS
Dependencies are organized in "groups" and each group can list one or more required modules, with an optional minimum version (or 0 for any version). The group name can be used in the DBIx::Class::Admin Modules required for the DBIx::Class administrative library o JSON::Any >= 1.23 o Moose >= 0.98 o MooseX::Types >= 0.21 o MooseX::Types::JSON >= 0.02 o MooseX::Types::LoadableClass >= 0.011 o MooseX::Types::Path::Class >= 0.05 o namespace::autoclean >= 0.09 Requirement group: admin dbicadmin Modules required for the CLI DBIx::Class interface dbicadmin o Getopt::Long::Descriptive >= 0.081 o JSON::Any >= 1.23 o Moose >= 0.98 o MooseX::Types >= 0.21 o MooseX::Types::JSON >= 0.02 o MooseX::Types::LoadableClass >= 0.011 o MooseX::Types::Path::Class >= 0.05 o Text::CSV >= 1.16 o namespace::autoclean >= 0.09 Requirement group: admin_script Storage::DBI::deploy() Modules required for "deploy" in DBIx::Class::Storage::DBI and "deployment_statements" in DBIx::Class::Storage::DBI o SQL::Translator >= 0.11016 Requirement group: deploy Sybase ASE support Modules required to connect to Sybase ASE o DBD::Sybase Requirement group: rdbms_ase DB2 support Modules required to connect to DB2 o DBD::DB2 Requirement group: rdbms_db2 DB2 on AS/400 support Modules required to connect to DB2 on AS/400 o DBD::ODBC Requirement group: rdbms_db2_400 Firebird support Modules required to connect to Firebird o DBD::Firebird Requirement group: rdbms_firebird Firebird support via DBD::InterBase Modules required to connect to Firebird via DBD::InterBase o DBD::InterBase Requirement group: rdbms_firebird_interbase Firebird support via DBD::ODBC Modules required to connect to Firebird via DBD::ODBC o DBD::ODBC Requirement group: rdbms_firebird_odbc Informix support Modules required to connect to Informix o DBD::Informix Requirement group: rdbms_informix MS Access support via DBD::ADO (Windows only) Modules required to connect to MS Access via DBD::ADO. This particular DBD is available on Windows only o DBD::ADO Requirement group: rdbms_msaccess_ado MS Access support via DBD::ODBC Modules required to connect to MS Access via DBD::ODBC o DBD::ODBC Requirement group: rdbms_msaccess_odbc MSSQL support via DBD::ADO (Windows only) Modules required to connect to MSSQL via DBD::ADO. This particular DBD is available on Windows only o DBD::ADO Requirement group: rdbms_mssql_ado MSSQL support via DBD::ODBC Modules required to connect to MSSQL via DBD::ODBC o DBD::ODBC Requirement group: rdbms_mssql_odbc MSSQL support via DBD::Sybase Modules required to connect to MSSQL via DBD::Sybase o DBD::Sybase Requirement group: rdbms_mssql_sybase MySQL support Modules required to connect to MySQL o DBD::mysql Requirement group: rdbms_mysql Oracle support Modules required to connect to Oracle o DBD::Oracle o Math::Base36 >= 0.07 o Math::BigInt >= 1.80 Requirement group: rdbms_oracle PostgreSQL support Modules required to connect to PostgreSQL o DBD::Pg Requirement group: rdbms_pg SQLAnywhere support Modules required to connect to SQLAnywhere o DBD::SQLAnywhere Requirement group: rdbms_sqlanywhere SQLAnywhere support via DBD::ODBC Modules required to connect to SQLAnywhere via DBD::ODBC o DBD::ODBC Requirement group: rdbms_sqlanywhere_odbc SQLite support Modules required to connect to SQLite o DBD::SQLite Requirement group: rdbms_sqlite Storage::Replicated Modules required for DBIx::Class::Storage::DBI::Replicated o Moose >= 0.98 o MooseX::Types >= 0.21 o MooseX::Types::LoadableClass >= 0.011 Requirement group: replicated METHODS
req_group_list Arguments: none Return Value: \%list_of_requirement_groups This method should be used by DBIx::Class packagers, to get a hashref of all dependencies keyed by dependency group. Each key (group name) can be supplied to one of the group-specific methods below. req_list_for Arguments: $group_name Return Value: \%list_of_module_version_pairs This method should be used by DBIx::Class extension authors, to determine the version of modules a specific feature requires in the current version of DBIx::Class. See the "SYNOPSIS" for a real-world example. req_ok_for Arguments: $group_name Return Value: 1|0 Returns true or false depending on whether all modules required by $group_name are present on the system and loadable. req_missing_for Arguments: $group_name Return Value: $error_message_string Returns a single line string suitable for inclusion in larger error messages. This method would normally be used by DBIx::Class core- module author, to indicate to the user that he needs to install specific modules before he will be able to use a specific feature. For example if some of the requirements for "deploy" are not available, the returned string could look like: SQL::Translator >= 0.11016 (see DBIx::Class::Optional::Dependencies for details) The author is expected to prepend the necessary text to this message before returning the actual error seen by the user. die_unless_req_ok_for Arguments: $group_name Checks if "req_ok_for" passes for the supplied $group_name, and in case of failure throws an exception including the information from "req_missing_for". req_errorlist_for Arguments: $group_name Return Value: \%list_of_loaderrors_per_module Returns a hashref containing the actual errors that occurred while attempting to load each module in the requirement group. AUTHOR
See "CONTRIBUTORS" in DBIx::Class. LICENSE
You may distribute this code under the same terms as Perl itself perl v5.18.2 2014-01-30 DBIx::Class::Optional::Dependencies(3)
All times are GMT -4. The time now is 03:58 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy