Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Unique entries in multiple files

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Unique entries in multiple files
# 8  
Old 09-02-2015
Quote:
Originally Posted by Aia
Could you explain how it doesn't work? I know why it doesn't, but I am quite sure is not the reason why you think it doesn't.

Perhaps, could you say why this, by itself, would not help?

Code:
DIR=/usr/src/count_test
FILES=$DIR/*
sed -n "s/.*user=\([^,]*\),.*/\1/p" $FILES | sort -u

It is printing data from file, instead of counting it.
# 9  
Old 09-02-2015
Well, would this solve your request better? Try:
Code:
sed -n "s/.*LOGIN.*user=\([^,]*\),.*/\1/p" $FILES | sort | uniq -c
      2 abc@example.com
      2 def@example.com
      2 mno@example.com

# 10  
Old 09-02-2015
Quote:
Originally Posted by RudiC
Well, would this solve your request better? Try:
Code:
sed -n "s/.*LOGIN.*user=\([^,]*\),.*/\1/p" $FILES | sort | uniq -c
      2 abc@example.com
      2 def@example.com
      2 mno@example.com

Not really. If I make a copy of the log file in the directory and run:
Code:
#!/bin/bash
DIR=/usr/src/count_test
FILES=$DIR/*
for i in $FILES
    do
 sed -n "s/.*LOGIN.*user=\([^,]*\),.*/\1/p" $FILES | sort | uniq -c
done

then I will get :
2 abc@example.com
2 def@example.com
2 mno@example.com
2 abc@example.com
2 def@example.com
2 mno@example.com
Which is not the case. should be:
4 abc@example.com
4 def@example.com
4 mno@example.com
# 11  
Old 09-02-2015
In the loop, don't use $FILES - use $i, the loop variable.

---------- Post updated at 12:41 ---------- Previous update was at 10:59 ----------

Sorry, my answer was incomplete. If you insist on the for loop, modify it like
Code:
for i in $FILES 
  do  sed -n "s/.*LOGIN.*user=\([^,]*\),.*/\1/p" $i 
  done | sort | uniq -c

This User Gave Thanks to RudiC For This Post:
ramirez987
# 12  
Old 09-11-2015
I have another problem.

I would like this script to look for "from=<" and "@example.com>" and count logins between theese two phrases. E.g.:

Code:
Something somethingso from=<abc@xgx.com> methingsomething somethingfrom=<ramirez987@example.com> somethingsomething something
Something somethingso from=<abc@xxx.com> methingsomething somethingfrom=<ramirez987@example.com> somethingsomething something
Something somethingso methingsomething somefrom=<abc@xxx.com> thingfrom=<alison@example.com> somethingsomething something

result:
1 alison
2 ramirez987
Last edited by ramirez987; 09-11-2015 at 10:32 AM..
# 13  
Old 09-13-2015
And you want the second occurrence in a line, not the first one? Are there always two?

---------- Post updated at 22:16 ---------- Previous update was at 22:15 ----------

Try
Code:
sed -n "s/^.*from=<\([^@]*\)@.*/\1/p" file | sort | uniq -c
      1 alison
      2 ramirez987

# 14  
Old 09-14-2015
Quote:
Originally Posted by RudiC
And you want the second occurrence in a line, not the first one? Are there always two?

---------- Post updated at 22:16 ---------- Previous update was at 22:15 ----------

Try
Code:
sed -n "s/^.*from=<\([^@]*\)@.*/\1/p" file | sort | uniq -c
      1 alison
      2 ramirez987

Hi,

There is no permanent place in which the address appeares, so it is not about "second occurrence". I just want to list all lolgins between the strings: "from=<" and "@example.com>"
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Split into multiple files by using Unique columns in a UNIX file

I have requirement to split below file (sample.csv) into multiple files by using the unique columns (first 3 are unique columns) sample.csv 123|22|56789|ABCDEF|12AB34|2019-07-10|2019-07-10|443.3400|1|1 123|12|5679|BCDEFG|34CD56|2019-07-10|2019-07-10|896.7200|1|2... (3 Replies)
Discussion started by: RVSP
3 Replies

2. Shell Programming and Scripting

Getting unique sequences from multiple fasta file

Hi, I have a fasta file with multiple sequences. How can i get only unique sequences from the file. For example my_file.fasta >seq1 TCTCAAAGAAAGCTGTGCTGCATACTGTACAAAACTTTGTCTGGAGAGATGGAGAATCTCATTGACTTTACAGGTGTGGACGGTCTTCAGAGATGGCTCAAGCTAACATTCCCTGACACACCTATAGGGAAAGAGCTAAC >seq2... (3 Replies)
Discussion started by: Ibk
3 Replies

3. Shell Programming and Scripting

Sort unique by multiple fields

i need to sort to get all the unique records based on the 1st and 2nd column, and keep the record with the highest value on 5th column if there are duplicates, every column with varies length a^2^x^y^z bxc^2xx2^aa^bvxxxx^cdd a^3^1^2^3 a^2^x^1^c I want a result which will only keep the 1st... (2 Replies)
Discussion started by: dtdt
2 Replies

4. Shell Programming and Scripting

Count Unique values from multiple lists of files

Looking for a little help here. I have 1000's of text files within a multiple folders. YYYY/ /MM /1000's Files Eg. 2014/01/1000 files 2014/02/1237 files 2014/03/1400 files There are folders for each year and each month, and within each monthly folder there are... (4 Replies)
Discussion started by: whegra
4 Replies

5. Shell Programming and Scripting

Reading multiple values from multiple lines and columns and setting them to unique variables.

Hello, I would like to ask for help with csh script. An example of an input in .txt file is below, the number of lines varies from file to file and I have 2 or 3 columns with values. I would like to read all the values (probably one by one) and set them to independent unique variables that... (7 Replies)
Discussion started by: FMMOLA
7 Replies

6. Shell Programming and Scripting

Compare columns of multiple files and print those unique string from File1 in an output file.

Hi, I have multiple files that each contain one column of strings: File1: 123abc 456def 789ghi File2: 123abc 456def 891jkl File3: 234mno 123abc 456def In total I have 25 of these type of file. (5 Replies)
Discussion started by: owwow14
5 Replies

7. Shell Programming and Scripting

Compare multiple files, identify common records and combine unique values into one file

Good morning all, I have a problem that is one step beyond a standard awk compare. I would like to compare three files which have several thousand records against a fourth file. All of them have a value in each row that is identical, and one value in each of those rows which may be duplicated... (1 Reply)
Discussion started by: nashton
1 Replies

8. Shell Programming and Scripting

Deriving unique entries from multiple repeating patterns

Dear All, I have a below one column data.(example) Col1 1 2 . . 25 8 9 25 1 2 . . 25 Where each entry(row) is a number from 1-25, but in place whereever mentioned with . we have all the entries 1-25, but some places where ever no . like in 8 9 25 I have only 3 entries. No I... (14 Replies)
Discussion started by: ks_reddy
14 Replies

9. Shell Programming and Scripting

Compare multiple files and print unique lines

Hi friends, I have multiple files. For now, let's say I have two of the following style cat 1.txt cat 2.txt output.txt Please note that my files are not sorted and in the output file I need another extra column that says the file from which it is coming. I have more than 100... (19 Replies)
Discussion started by: jacobs.smith
19 Replies

10. Shell Programming and Scripting

adding a column of unique string to multiple files

Hi, I need to add a column of unique string to individual csv file. my individual csv file looks like this and each file has variable row length: 178.52,39.4,train,0.003355544375334351,39.15752753933254,0.4895933968953914... (7 Replies)
Discussion started by: ida1215
7 Replies
Login or Register to Ask a Question