Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Counting similar lines from file UNIX

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Counting similar lines from file UNIX
# 1  
Old 08-23-2009
Lightbulb Counting similar lines from file UNIX
I have a file which contains data as below:

Code:
nbk1j7o pageName=/jsp/RMBS/RMBSHome.jsf
nbk1j7o pageName=/jsp/RMBS/RMBSHome.jsf
nbk1j7o pageName=/jsp/RMBS/RMBSHome.jsf
nbk1j7o pageName=/jsp/RMBS/RMBSHome.jsf
nbk1j7o pageName=/jsp/common/index.jsf
nbk1j7o pageName=/jsp/common/index.jsf
nbk1wqe pageName=/jsp/RMBS/RMBSHome.jsf
nbk1wqe pageName=/jsp/common/index.jsf
nbk2coz pageName=/jsp/RMBS/PassThrough.jsf
nbk2coz pageName=/jsp/RMBS/PassThrough.jsf
nbk2coz pageName=/jsp/RMBS/PassThrough.jsf
nbk2coz pageName=/jsp/RMBS/PassThrough.jsf
nbk2coz pageName=/jsp/RMBS/PassThrough.jsf
nbk2coz pageName=/jsp/RMBS/PassThrough.jsf
nbk2coz pageName=/jsp/RMBS/PassThrough.jsf

I want output like:

--------------------------------------------------------
Code:
NBKID       PAGE ACCESSED                             COUNT
--------------------------------------------------------
nbk1j7o    pageName=/jsp/RMBS/RMBSHome.jsf    4
nbk1j7o    pageName=/jsp/common/index.jsf        2
nbk1wqe  pageName=/jsp/RMBS/RMBSHome.jsf     1
nbk1wqe  pageName=/jsp/common/index.jsf         1
nbk2coz   pageName=/jsp/RMBS/PassThrough.jsf   7



In short, i want to count the similar lines and remove multiple entries and include the count of that particular line.
Last edited by vgersh99; 08-23-2009 at 07:42 PM.. Reason: code tags, PLEASE!
# 2  
Old 08-23-2009
To keep the forums high quality for all users, please take the time to format your posts correctly.
  1. Use Code Tags when you post any code or data samples so others can easily read your code.
    You can easily do this by highlighting your code and then clicking on the # in the editing menu. (You can also type code tags [code] and [/code] by hand.)
  2. Avoid adding color or different fonts and font size to your posts.
    Selective use of color to highlight a single word or phrase can be useful at times, but using color, in general, makes the forums harder to read, especially bright colors like red.
  3. Be careful when you cut-and-paste, edit any odd characters and make sure all links are working property.

Thank You.

The UNIX and Linux Forums
Reply With Quote


Code:
awk '{a[$0]++}END{for(i in a)print i,a[i]}' file | sort

# 3  
Old 08-23-2009
A slightly different take on the problem, where counting of occurrences is done in the shell itself:

Code:
sort data.txt | uniq -c | perl -ne 'split; printf("%s %s %s\n",$_[1],$_[2],$_[0])'

tyler_durden
# 4  
Old 08-23-2009
Thanks
Thanks for the help.. but the output is little unformatted, I wanted these values in seperated column so that they look good...
# 5  
Old 08-23-2009
Quote:
Originally Posted by mohsin.quazi
Thanks for the help.. but the output is little unformatted, I wanted these values in seperated column so that they look good...
Please define 'looking good'. Please use code tags when posting data/code samples.

---------- Post updated at 05:41 PM ---------- Previous update was at 05:41 PM ----------

To keep the forums high quality for all users, please take the time to format your posts correctly.

First of all, use Code Tags when you post any code or data samples so others can easily read your code. You can easily do this by highlighting your code and then clicking on the # in the editing menu. (You can also type code tags [code] and [/code] by hand.)

Second, avoid adding color or different fonts and font size to your posts. Selective use of color to highlight a single word or phrase can be useful at times, but using color, in general, makes the forums harder to read, especially bright colors like red.

Third, be careful when you cut-and-paste, edit any odd characters and make sure all links are working property.

Thank You.

The UNIX and Linux Forums
# 6  
Old 08-24-2009
the looking good version,lol
Code:
sort -t= -k2 file | uniq -c | awk '{printf "%s %-40s%d\n",$2,$3,$1}'

# 7  
Old 08-25-2009
Code:
 awk '{_[$0]++}
        END{
        for(i in _){
        print i" "_[i]
        }
        }'

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Reducing text file using similar lines

Hello, I am a java programmer but want to try unix for a purpose where I need to reduce a file using its first field.. Here is the sample data: admin;2;0;; admission;8;0;; aman;1;0;; caroline;0;4;; cook;0;4;; cook;2;0;; far;0;3;; far;1;5;; I am explaining the dataset first. There... (5 Replies)
Discussion started by: shekhar2010us
5 Replies

2. Shell Programming and Scripting

Help in unix script to join similar lines of input

Hi, I have been thinking of how to script this but i have no clue at all.. Could someone please help me out or give me some idea on this? I would like to group those lines with the same first variable in each line, joining the 2nd variables with commas. Let's say i have the following input. ... (3 Replies)
Discussion started by: rei125
3 Replies

3. Shell Programming and Scripting

removing lines with similar values from file

Hello, got a file with this structure: 33274 171030 02/29/2012 37897 P_GEH 2012-02-29 10:31:26 33275 171049 02/29/2012 38132 P_GEH 2012-02-29 10:35:27 33276 171058 02/29/2012 38515 P_GEH 2012-02-29 10:43:26 33277 170748 02/29/2012 40685 P_KOM ... (3 Replies)
Discussion started by: krecik28
3 Replies

4. Shell Programming and Scripting

extracting lines from a file with similar first name

consider i have two files cat onlyviews1.sql CREATE VIEW V11 AS SELECT id, name, FROM etc etc WHERE etc etc; CREATE VIEW V22 AS SELECT id, name, FROM etc etc WHERE etc etc; CREATE VIEW V33 AS (10 Replies)
Discussion started by: vivek d r
10 Replies

5. UNIX for Dummies Questions & Answers

Matching and reporting near-similar lines in a file

Hi, I have a file with the lines as below: C_10_A05_T7 C_10_A06_SP6 C_10_B05_SP6 C_10_B05_T7 C_10_B01_SP6 C_10_B01_T7 C_12_G07_SP6 C_12_G11_SP6 C_12_G11_T7 C_2_H18_T7 C_2_I02_SP6 C_2_I02_T7 C_2_I13_SP6 C_2_I17_SP6 The four segments of each line are connected by '_' symbols. I... (7 Replies)
Discussion started by: Fahmida
7 Replies

6. Shell Programming and Scripting

remove one of each similar lines in a file

Hello folks I have a question for you gurus of sed or grep (maybe awk, but I would prefer the first two) I have a file (f1) that says: (actually, these are not numbers but md5sum, but for simplicity, let's assume these numbers.) 1 2 3 4 5And I have a file (f2) that says 1|a 1|b 1|c 2|d... (3 Replies)
Discussion started by: tukuyomi
3 Replies

7. Shell Programming and Scripting

Counting similar lines

Hi, I have a little problem with counting lines. I know similar topics from this forum, but they don't resolve my problem. I have file with lines like this: 2009-05-25 16:55:32,143 some text some regular expressions ect. 2009-05-25 16:55:32,144 some text. 2009-05-28 18:15:12,148 some... (4 Replies)
Discussion started by: marcinnnn
4 Replies

8. Infrastructure Monitoring

Remove Similar Lines from a File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead... (4 Replies)
Discussion started by: Nysif Steve
4 Replies

9. Shell Programming and Scripting

How to sort a file and then print similar lines once

Hi! I have a trouble with the sort and the uniq. I know I have to use them, I just have trouble with putting them in the right order. I have a text file with unsorted lines (each line has a few words, the first word in the line is a number.). I need to sort this file in order to be... (6 Replies)
Discussion started by: shira
6 Replies

10. Shell Programming and Scripting

awk - Counting number of similar lines

Hi All I have the input file OMAK_11. OMAK 000002EXCLUDE 1341 OMAK 000002EXCLUDE 1341 OMAK 000002EXCLUDE 1341 OMAK 000003EXCLUDE 1341 OMAK 000003EXCLUDE 1341 OMAK 000003EXCLUDE ... (8 Replies)
Discussion started by: dhanamurthy
8 Replies
Login or Register to Ask a Question