Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: awk - mixed for and if to select particular lines in a data file
Top Forums Shell Programming and Scripting awk - mixed for and if to select particular lines in a data file Post 302829657 by Don Cragun on Friday 5th of July 2013 01:00:27 PM
Old 07-05-2013
The following produces the output you requested but the order of the output is unspecified:
Code:
awk '
NF < 2 {next
}
!($1 in m) || m[$1] > $2 {
        m[$1] = $2
        o[$1] = $0
}
END {   for(i in o) print o[i]
}' input.txt

If you are using a Solaris system, use /usr/xpg4/bin/awk, /usr/xpg6/bin/awk, or nawk instead of /usr/bin/awk or /bin/awk.

Your sample has all of the input with a given 1st column value grouped together, but your statement of requirements didn't say anything about this. The code above accepts input in any order.

If your input always has all lines with the same 1st column value on adjacent input lines, this script can be rewritten to produce output when the 1st column value changes. This would take fewer resources for large input files and would produce output in the same order as the input.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Script needed to select and delete lower case and mixed case records

HELLO ALL, URGENTLY NEEDED A SCRIPT TO SELECT AND DELETE LOWER AND MIXED CASE RECORDS FROM A COLUMN IN A TABLE. FOR EXAMPLE : Table name is EMPLOYEE and the column name is CITY and the CITY column records will be: Newyork washington ... (1 Reply)
Discussion started by: abhilash mn
1 Replies

2. Shell Programming and Scripting

Select some lines from a txt file and create a new file with awk

Hi there, I have a text file with several colums separated by "|;#" I need to search the file extracting all columns starting with the value of "1" or "2" saving in a separate file just the first 7 columns of each row maching the criteria, with replacement of the saparators in the nearly created... (4 Replies)
Discussion started by: capnino
4 Replies

3. Shell Programming and Scripting

Select lines in which column have value greater than some percent of total file lines

i have a file in following format 1 32 3 4 6 4 4 45 1 45 4 61 54 66 4 5 65 51 56 65 1 12 32 85 now here the total number of lines are 8(they vary each time) Now i want to select only those lines in which the values... (6 Replies)
Discussion started by: vaibhavkorde
6 Replies

4. Shell Programming and Scripting

Extracting specific lines of data from a file and related lines of data based on a grep value range?

Hi, I have one file, say file 1, that has data like below where 19900107 is the date, 19900107 12 144 129 0.7380047 19900108 12 168 129 0.3149017 19900109 12 192 129 3.2766666E-02 ... (3 Replies)
Discussion started by: Wynner
3 Replies

5. Shell Programming and Scripting

select lines with certain values on certain fields with awk

I need a awk command to select from a log-file only the lines that have on the 2nd field (considering "|" separator) one of the values 10.216.22.XX or 10.216.22.YY or 10.216.22.ZZ and on the 4th field only values that contain strictly digits. I want the command to work parsing the file only once (I... (2 Replies)
Discussion started by: black_fender
2 Replies

6. UNIX for Advanced & Expert Users

Sort mixed data file

I have a text file and each field is separated by semicolon ( ; ). Field number 7 is internally separated by comma ( , ) and pipe ( | ) symbol. I want to sort file based on three different fields which are marked in BOLD. Here first BOLD field will have numbers upto the length of 9 characters,... (6 Replies)
Discussion started by: jnrohit2k
6 Replies

7. Shell Programming and Scripting

Using awk to parse a file with mixed formats in columns

Greetings I have a file formatted like this: rhino grey weight=1003;height=231;class=heaviest;histology=9,0,0,8 bird white weight=23;height=88;class=light;histology=7,5,1,0,0 turtle green weight=40;height=9;class=light;histology=6,0,2,0... (2 Replies)
Discussion started by: Twinklefingers
2 Replies

8. Shell Programming and Scripting

Select only the lines of a file starting with a field which is matcing a list. awk?

Hello I have a large file1 which has many events like "2014010420" and following lines under each event that start with text . It has this form: 2014010420 num --- --- num .... NTE num num --- num... EFA num num --- num ... LASW num num --- num... (9 Replies)
Discussion started by: phaethon
9 Replies

9. Shell Programming and Scripting

awk to select lines with maximum value of each record based on column value

Hello, I want to get the maximum value of each record separated by empty line based on the 3rd column of each row within each record? Input: A1 chr5D 634 7 82 707 A2 chr5D 637 6 82 713 A3 chr5D 637 5 82 713 A4 chr5D 626 1 82 704... (4 Replies)
Discussion started by: yifangt
4 Replies

10. Shell Programming and Scripting

awk to select 2D data bins

I wish to use AWK to do something akin: Select all 2D data with 1<$1<2 and -7.5<$2<-6.5 But it's not working awk 'END {print ($1<=2&&$1>=1&&$2<=-6.5&&$2>=-7.5)}' bla Data: -1.06897 -8.04482 -61.469 -1.13613 -8.04482 -61.2271 -1.00182 -8.04482 -61.2081 -1.06897 -8.13518 -60.8544... (2 Replies)
Discussion started by: chrisjorg
2 Replies

LEARN ABOUT CENTOS

largefile

largefile(5)                                            Standards, Environments, and Macros                                           largefile(5)

NAME

       largefile - large file status of utilities

DESCRIPTION

       A  large file is a regular file whose size is greater than or equal to 2 Gbyte ( 2**31 bytes). A small file is a regular file whose size is
       less than 2 Gbyte.

   Large file aware utilities
       A utility is called large file aware if it can process large files in the same manner as it does small files. A utility that is large  file
       aware is able to handle large files as input and generate as output large files that are being processed. The exception is where additional
       files  are used as system configuration files or support files that can augment the processing. For example, the file utility supports  the
       -m  option  for  an  alternative "magic" file and the -f option for a support file that can contain a list of file names. It is unspecified
       whether a utility that is large file aware will accept configuration or support files that are large files. If a large file  aware  utility
       does  not accept configuration or support files that are large files, it will cause no data loss or corruption upon encountering such files
       and will return an appropriate error.

       The following /usr/bin utilities are large file aware:

       adb           awk           bdiff         cat           chgrp
       chmod         chown         cksum         cmp           compress
       cp            csh           csplit        cut           dd
       dircmp        du            egrep         fgrep         file
       find          ftp           getconf       grep          gzip
       head          join          jsh           ksh           ln
       ls            mdb           mkdir         mkfifo        more
       mv            nawk          page          paste         pathchck
       pg            rcp           remsh         rksh          rm
       rmdir         rsh           sed           sh            sort
       split         sum           tail          tar           tee
       test          touch         tr            uncompress    uudecode
       uuencode      wc            zcat

       The following /usr/xpg4/bin utilities are large file aware:

       awk           cp            chgrp         chown         du
       egrep         fgrep         file          grep          ln
       ls            more          mv            rm            sed
       sh            sort          tail          tr

       The following /usr/xpg6/bin utilities are large file aware:

       getconf       ls            tr

       The following /usr/sbin utilities are large file aware:

       install       mkfile        mknod         mvdir         swap

       See the USAGE section of the swap(1M) manual page for limitations of swap on block devices greater than 2 Gbyte on a 32-bit operating  sys-
       tem.

       The following /usr/ucb utilities are large file aware:

       chown         from          ln            ls            sed
       sum           touch

       The /usr/bin/cpio and /usr/bin/pax utilities are large file aware, but cannot archive a file whose size exceeds 8 Gbyte - 1 byte.

       The /usr/bin/truss utilities has been modified to read a dump file and display information relevant to large files, such as offsets.

   cachefs file systems
       The following /usr/bin utilities are large file aware for cachefs file systems:

       cachefspack      cachefsstat

       The following /usr/sbin utilities are large file aware for cachefs file systems:

       cachefslog       cachefswssize   cfsadmin         fsck
       mount            umount

   nfs file systems
       The following utilities are large file aware for nfs file systems:

       /usr/lib/autofs/automountd    /usr/sbin/mount
       /usr/lib/nfs/rquotad

   ufs file systems
       The following /usr/bin utility is large file aware for ufs file systems:

              df

       The following /usr/lib/nfs utility is large file aware for ufs file systems:

              rquotad

       The following /usr/xpg4/bin utility is large file aware for ufs file systems:

              df

       The following /usr/sbin utilities are large file aware for ufs file systems:

       clri          dcopy         edquota       ff            fsck
       fsdb          fsirand       fstyp         labelit       lockfs
       mkfs          mount         ncheck        newfs         quot
       quota         quotacheck    quotaoff      quotaon       repquota
       tunefs        ufsdump       ufsrestore    umount

   Large file safe utilities
       A  utility  is called large file safe if it causes no data loss or corruption when it encounters a large file. A utility that is large file
       safe is unable to process properly a large file, but returns an appropriate error.

       The following /usr/bin utilities are large file safe:

       audioconvert     audioplay    audiorecord    comm          diff
       diff3            diffmk       ed             lp            mail

       mailcompat       mailstats    mailx          pack          pcat
       red              rmail        sdiff          unpack        vi
       view

       The following /usr/xpg4/bin utilities are large file safe:

       ed            vi            view

       The following /usr/xpg6/bin utility is large file safe:

       ed

       The following /usr/sbin utilities are large file safe:

       lpfilter      lpforms

       The following /usr/ucb utilities are large file safe:

       Mail          lpr

       The following /usr/lib utility is large file safe:

              sendmail

SEE ALSO

       lf64(5), lfcompile(5), lfcompile64(5)

SunOS 5.10                                                          7 Nov 2003                                                        largefile(5)
All times are GMT -4. The time now is 11:07 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy