Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Help around sort

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help around sort
# 1  
Old 07-16-2010
Help around sort
Hi.
I'm making some manipulations around the httpd.log.

Here's the commande line I'm using at this time :

Code:
cat /thewaytologs/log/httpd_access_log | awk '{print $3,$1}' | sort -k2,2 | uniq -c | sort -gr > /home/myhome/top_IP.txt

In result, i got something like that :

Quote:
9635 uid=nameA,ou=people,dc=univ,dc=fr his-IP-at-this-Time
9142 uid=nameB,ou=people,dc=univ,dc=fr his-IP-at-this-Time
8316 uid=nameC,ou=people,dc=univ,dc=fr his-IP-at-this-Time
7691 uid=nameA,ou=people,dc=univ,dc=fr his-IP-at-this-Time
5665 uid=nameA,ou=people,dc=univ,dc=fr his-IP-at-this-Time
4300 uid=nameB,ou=people,dc=univ,dc=fr his-IP-at-this-Time
3886 uid=nameB,ou=people,dc=univ,dc=fr his-IP-at-this-Time
3701 uid=nameA,ou=people,dc=univ,dc=fr his-IP-at-this-Time
3534 uid=nameC,ou=people,dc=univ,dc=fr his-IP-at-this-Time
3267 uid=nameC,ou=people,dc=univ,dc=fr his-IP-at-this-Time
3125 uid=nameC,ou=people,dc=univ,dc=fr his-IP-at-this-Time
3024 uid=nameC,ou=people,dc=univ,dc=fr his-IP-at-this-Time
3016 uid=nameA,ou=people,dc=univ,dc=fr his-IP-at-this-Time
2961 uid=nameD,ou=people,dc=univ,dc=fr his-IP-at-this-Time
But I'd like to have something like that :

Quote:
uid=nameA,ou=people,dc=univ,dc=fr
his-IP-at-this-Time
his-IP-at-this-Time (if different)
his-IP-at-this-Time (if different)
uid=nameB,ou=people,dc=univ,dc=fr
his-IP-at-this-Time
his-IP-at-this-Time (if different)
his-IP-at-this-Time (if different)
uid=nameC,ou=people,dc=univ,dc=fr
his-IP-at-this-Time
his-IP-at-this-Time (if different)
his-IP-at-this-Time (if different)
uid=nameD,ou=people,dc=univ,dc=fr
his-IP-at-this-Time
his-IP-at-this-Time (if different)
his-IP-at-this-Time (if different)
Help welcome.
Thanks
Best regards
D.

---------- Post updated at 10:26 AM ---------- Previous update was at 09:35 AM ----------

Of course, any other suggestion to get the stuff working (with some other sort of script) would be welcome ;-)
# 2  
Old 07-16-2010
Could you please post a relevant part of your original log file.
# 3  
Old 07-16-2010
Yep :

Quote:
INCOMING IP - uid=jroux,ou=people,dc=univ-myuniv,dc=fr [12/Jul/2010:15:03:14 +0200] "GET /anURL" Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6"
INCOMING IP - uid=arousseau,ou=people,dc=univ-myuniv,dc=fr [12/Jul/2010:15:34:11 +0200] "GET /anURL" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6"
and so on (all lines on that structure)

Thanks
# 4  
Old 07-16-2010
Assuming the identity information doesn't contain white space characters:
Code:
awk 'END { 
  for (U in uid) printf "%s\n%s\n\n", U, uid[U] 
  }
{ uid[$3] = uid[$3] ? uid[$3] RS $1 : $1 }
  ' logfile

Use gawk, nawk or /usr/xpg4/bin/awk on Solaris.
# 5  
Old 07-16-2010
waouhh - gonna test it just now
# 6  
Old 07-16-2010
radoulov's code can't remove duplicate records.

Code:
$ cat logfile
10.1.2.3 - uid=jroux,ou=people,dc=univ-myuniv,dc=fr [12/Jul/2010:15:03:14 +0200] "GET /anURL" Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6"
10.1.2.3 - uid=jroux,ou=people,dc=univ-myuniv,dc=fr [12/Jul/2010:15:03:14 +0200] "GET /anURL" Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6"
10.2.3.4 - uid=arousseau,ou=people,dc=univ-myuniv,dc=fr [12/Jul/2010:15:34:11 +0200] "GET /anURL" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6"
10.4.3.4 - uid=arousseau,ou=people,dc=univ-myuniv,dc=fr [12/Jul/2010:15:34:11 +0200] "GET /anURL" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6"
10.4.3.4 - uid=arousseau,ou=people,dc=univ-myuniv,dc=fr [12/Jul/2010:15:34:11 +0200] "GET /anURL" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6"
10.5.3.4 - uid=arousseau,ou=people,dc=univ-myuniv,dc=fr [12/Jul/2010:15:34:11 +0200] "GET /anURL" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6"
10.4.3.4 - uid=arousseau,ou=people,dc=univ-myuniv,dc=fr [12/Jul/2010:15:34:11 +0200] "GET /anURL" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6"

Code:
$ awk 'END {
   for (U in uid) printf "%s\n%s\n\n", U, uid[U]
   }
 { uid[$3] = uid[$3] ? uid[$3] RS $1 : $1 }
   ' logfile

uid=arousseau,ou=people,dc=univ-myuniv,dc=fr
10.2.3.4
10.4.3.4
10.4.3.4
10.5.3.4
10.4.3.4

uid=jroux,ou=people,dc=univ-myuniv,dc=fr
10.1.2.3
10.1.2.3

Try mine:
Code:
 awk '!a[$1 FS $3] {b[$3]=b[$3] FS $1;a[$1 FS $3]=1} 
      END {for (i in b) {print i;split(b[i],s," "); for (j in s) print s[j]}}' logfile


uid=arousseau,ou=people,dc=univ-myuniv,dc=fr
10.2.3.4
10.4.3.4
10.5.3.4
uid=jroux,ou=people,dc=univ-myuniv,dc=fr
10.1.2.3

# 7  
Old 07-16-2010
Good point, I didn't think about that!

Another version:

Code:
awk 'END { 
  for (U in uid) {
    print U
    for (IP in ip)
      if ( (IP, U) in uip )
        print IP
    }    
  }
{  
  uip[$1, $3]; uid[$3]; ip[$1] 
  }' logfile
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Use sort to sort numerical column

How to sort the following output based on lowest to highest BE? The following sort does not work. $ sort -t. -k1,1n -k2,2n bfd.txt BE31.116 0s 0s DOWN DAMP BE31.116 0s 0s DOWN DAMP BE31.117 0s 0s ... (7 Replies)
Discussion started by: sand1234
7 Replies

2. UNIX for Beginners Questions & Answers

Difference of Sort -n -k2 -k3 & Sort -n -k2,3

Hi, Could anyone kindly show me a link or explain the difference between sort -n -k2 -k3 & sort -n -k2,3 Also, if I like to remove the row with repetition at both $2 and $3, Can I safely use sort -u -k2 -k3 Example; 100 20 30 100 20 30 So, both $2 and $3 are same and I... (2 Replies)
Discussion started by: Indra2011
2 Replies

3. Shell Programming and Scripting

Sort help: How to sort collected 'file list' by date stamp :

Hi Experts, I have a filelist collected from another server , now want to sort the output using date/time stamp filed. - Filed 6, 7,8 are showing the date/time/stamp. Here is the input: #---------------------------------------------------------------------- -rw------- 1 root ... (3 Replies)
Discussion started by: rveri
3 Replies

4. Shell Programming and Scripting

Help with sort word and general numeric sort at the same time

Input file: 100%ABC2 3.44E-12 USA A2M%H02579 0E0 UK 100%ABC2 5.34E-8 UK 100%ABC2 3.25E-12 USA A2M%H02579 5E-45 UK Output file: 100%ABC2 3.44E-12 USA 100%ABC2 3.25E-12 USA 100%ABC2 5.34E-8 UK A2M%H02579 0E0 UK A2M%H02579 5E-45 UK Code try: sort -k1,1 -g -k2 -r input.txt... (2 Replies)
Discussion started by: perl_beginner
2 Replies

5. Shell Programming and Scripting

Alternate to sort --random-sort

sort --random-sort The full command is path=`find /testdir -maxdepth 1 -mindepth 1 -type d | ***Some sort of sort function*** | head -1` I have a list I want to randomly sort. It works fine in ubuntu but on a 'osx lion' sort dosen't have the --random-sort option. I don't want to... (5 Replies)
Discussion started by: digitalviking
5 Replies

6. Shell Programming and Scripting

Help to sort out... Possible use of sort command

I have an input like 4.3.6.66 4.3.6.67 4.3.6.70 4.3.6.25 4.3.6.15 4.3.6.54 4.3.6.44 4.3.6.34 4.3.6.24 4.3.6.14 4.3.6.53 4.3.6.43 4.3.6.49 4.3.6.33 4.3.6.52 4.3.6.19 4.3.6.58 4.3.6.42 (5 Replies)
Discussion started by: dnam9917
5 Replies

7. Shell Programming and Scripting

Is it Possible to sort a list of hexadecimal numbers using "sort" command?

Hello Everybody :) !!!. i have question in mind, is it possible to sort a list of hexadecimal numbers using "sort" command? (9 Replies)
Discussion started by: Kesavan
9 Replies

8. UNIX for Advanced & Expert Users

Script to sort the files and append the extension .sort to the sorted version of the file

Hello all - I am to this forum and fairly new in learning unix and finding some difficulty in preparing a small shell script. I am trying to make script to sort all the files given by user as input (either the exact full name of the file or say the files matching the criteria like all files... (3 Replies)
Discussion started by: pankaj80
3 Replies

9. Shell Programming and Scripting

How to Sort Floating Numbers Using the Sort Command?

Hi to all. I'm trying to sort this with the Unix command sort. user1:12345678:3.5:2.5:8:1:2:3 user2:12345679:4.5:3.5:8:1:3:2 user3:12345687:5.5:2.5:6:1:3:2 user4:12345670:5.5:2.5:5:3:2:1 user5:12345671:2.5:5.5:7:2:3:1 I need to get this: user3:12345687:5.5:2.5:6:1:3:2... (7 Replies)
Discussion started by: daniel.gbaena
7 Replies

10. Shell Programming and Scripting

sort

I check the man page but I still cannot see what this command is supposed to do sort +5 -6 <file> It just seems to sort the file as normal?? Thanks Calypso (3 Replies)
Discussion started by: Calypso
3 Replies
Login or Register to Ask a Question