Text Proccessing with sort,uniq,awk Post: 302482601

Sponsored Content

Top Forums Shell Programming and Scripting Text Proccessing with sort,uniq,awk Post 302482601 by anurag.singh on Wednesday 22nd of December 2010 04:37:10 AM

12-22-2010

Registered User

1. $1 != "X" == >> $1 is 1st field of every record, This check ensures that we should not process 1st record in inputFIle, The Header Record (X , ID , Date, Time, Y
). But this check is not necessary(And can be removed) in post #10 command. 2nd if condition (check on $4) will not be true for Header record and so that will not be printed.
2. what does !a[$3] stand for? == >> This checks if a[$3] is NULL OR has any value set in it. if(!a[$3]) is equivalent to if(a[$3]==NULL)
3. substr($3,6,5) == >> $3 will have year value like 2010-12-03. String indices in awk starts at 1. To get 12-03, start index is 6 (12-03 starts from 6th character) and then it goes upto the end i.e. lenght is 5.
substr($3,6) also gives same result. length is not needed if substring goes upto the last character.

I learned most of stuffs here only (After building few basics) by following posts of experts here Scrutinizer, vgersh99, Franklin52, fpmurphy, scottn, radoulov etc to name a few

Few AWK articles (Many more can be found on internet):
opengroup_awk
grymoire
Utrecht_University_docs
thegeekstuff_awk_with_examples (link removed)

Last edited by anurag.singh; 12-22-2010 at 09:01 AM.. Reason: typo

This User Gave Thanks to anurag.singh For This Post:

anurag.singh

View Public Profile for anurag.singh

Find all posts by anurag.singh

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

sort/uniq

I have a file: Fred Fred Fred Jim Fred Jim Jim If sort is executed on the listed file, shouldn't the output be?: Fred Fred Fred Fred Jim Jim Jim

2. UNIX for Dummies Questions & Answers

Help with Last,uniq, sort and cut

Using the last, uniq, sort and cut commands, determine how many times the different users have logged in. I know how to use the last command and cut command... i came up with last | cut -f1 -d" " | uniq i dont know if this is right, can someone please help me... thanks

3. Shell Programming and Scripting

sort and uniq in perl

Does anyone have a quick and dirty way of performing a sort and uniq in perl? How an array with data like: this is bkupArr BOLADVICE_VN this is bkupArr MLT6800PROD2A this is bkupArr MLT6800PROD2A this is bkupArr BOLADVICE_VN_7YR this is bkupArr MLT6800PROD2A I want to sort it...

4. Shell Programming and Scripting

Sort, Uniq, Duplicates

Input File is : ------------- 25060008,0040,03, 25136437,0030,03, 25069457,0040,02, 80303438,0014,03,1st 80321837,0009,03,1st 80321977,0009,03,1st 80341345,0007,03,1st 84176527,0047,03,1st 84176527,0047,03, 20000735,0018,03,1st 25060008,0040,03, I am using the following in the script...

5. Shell Programming and Scripting

Help with Uniq and sort

The key is first field i want only uniq record for the first field in file. I want the output as or output as Appreciate help on this

6. Shell Programming and Scripting

Sort and uniq after comparision

Hi All, I have a text file with the format shown below. Some of the records are duplicated with the only exception being date (Field 15). I want to compare all duplicate records using subscriber number (field 7) and keep only those records with greater date. ...

7. Shell Programming and Scripting

Sort uniq or awk

Hi again, I have files with the following contents datetime,ip1,port1,ip2,port2,number How would I find out how many times ip1 field shows up a particular file? Then how would I find out how many time ip1 and port 2 shows up? Please mind the file may contain 100k lines.

8. Shell Programming and Scripting

File comparison and proccessing using awk

Hi Guys, I am having two requirement in one of my scripts. please help out to find a fast solution using AWK (since there is lot of data to be processed) 1) First snippet - File1 has two columns and file2 has three columns If any value of column 1 of file1 matches with column 1...

9. Shell Programming and Scripting

Uniq or sort -u or similar only between { }

Hi ! I am trying to remove doubbled entrys in a textfile only between delimiters. Like that example but i dont know how to do that with sort or similar. input: { aaa aaa } { aaa aaa } output: { aaa } {

10. UNIX for Dummies Questions & Answers

Uniq and sort -u

Hello all, Need to pick your brains, I have a 10Gb file where each row is a name, I am expecting about 50 names in total. So there are a lot of repetitions in clusters. So I want to do a sort -u file Will it be considerably faster or slower to use a uniq before piping it to sort...

LEARN ABOUT CENTOS

paps

PAPS(1) 						      General Commands Manual							   PAPS(1)

NAME

       paps - UTF-8 to PostScript converter using Pango

SYNOPSIS

       paps [options] files...

DESCRIPTION

       paps reads a UTF-8 encoded file and generates a PostScript language rendering of the file. The rendering is done by creating outline curves
       through the pango ft2 backend.

OPTIONS

       These programs follow the usual GNU command line syntax, with long options starting with  two  dashes  (`-').   A  summary  of  options	is
       included below.

       --landscape
	      Landscape output. Default is portrait.

       --columns=cl
	      Number of columns output. Default is 1.

       --font=desc
	      Set the font description. Default is Monospace 12.

       --rtl  Do rtl layout.

       --paper ps
	      Choose paper size. Known paper sizes are legal, letter, a4. Default is A4.

       --bottom-margin=bm
	      Set bottom margin in postscript points (1/72 inch). Default is 36.

       --top-margin=tm
	      Set top margin. Default is 36.

       --left-margin=lm
	      Set left margin. Default is 36.

       --right-margin=rm
	      Set right margin. Default is 36.

       --help Show summary of options.

       --header
	      Draw page header for each page.

       --markup
	      Interpret the text as pango markup.

       --encoding=ENCODING
	      Assume the documentation encoding is ENCODING.

       --lpi  Set the lines per inch. This determines the line spacing.

       --cpi  Set the characters per inch. This is an alternative method of specifying the font size.

       --stretch-chars
	      Indicates  that  characters  should be stretched in the y-direction to fill up their vertical space. This is similar to the texttops
	      behaviour.

AUTHOR

       paps was written by Dov Grobgeld <dov.grobgeld@gmail.com>.

       This manual page was written by Lior Kaplan <kaplan@debian.org>, for the Debian project (but may be used by others).

								  April  17, 2006							   PAPS(1)

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

sort/uniq

Discussion started by: jimmyflip

2. UNIX for Dummies Questions & Answers

Help with Last,uniq, sort and cut

Discussion started by: jay1228

3. Shell Programming and Scripting

sort and uniq in perl

Discussion started by: reggiej

4. Shell Programming and Scripting

Sort, Uniq, Duplicates

Discussion started by: Amruta Pitkar