Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Count Unique values from multiple lists of files
Top Forums Shell Programming and Scripting Count Unique values from multiple lists of files Post 302915732 by whegra on Thursday 4th of September 2014 12:07:20 PM
Old 09-04-2014
My input looks like this

Files
Code:
20140902114511523AAA112.txt
20140902114511523BBB113.txt
20140902114511523XXX114.txt

Within each file there values look like this.
Code:
01650126861409662618TUCAAAA_____#20140902075659
01650018671409678351TUCABBB_____#20140902121932
01650070281409691297TUCARSA_____#20140902155457
01650133911409663109TUCAPDS_____#20140902155457

Initially was i just going to cat the files and grep the out put and count

eg. cat *.txt|grep TUCAAAA|wc -l for each folder and year.

I know there must be a much simpler way then pasting in hundreds of count lines into a .sh script and running that.

Another thought i had was to get a unique values, then do something like this.

Code:
while read line ; do
        a=`cat /home/test/2014/01/*.txt|grep $line|wc -l`
        echo $line $a >> /home/test/counts.txt
done < /home/test/values.txt

It's getting the unique values in the first place that is holding me up.
Last edited by Franklin52; 09-04-2014 at 02:43 PM.. Reason: added code tags
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to count Unique Values from a file.

Hi I have the following info in a file - <Cell id="25D"/> <Cell id="26A"/> <Cell id="26B"/> <Cell id="26C"/> <Cell id="27A"/> <Cell id="27B"/> <Cell id="27C"/> <Cell id="28A"/> I would like to know how would you go about counting all... (4 Replies)
Discussion started by: Prega
4 Replies

2. Shell Programming and Scripting

List unique values and count instances in .csv file

I need to take the second column of a .csv file and count the number of instances of each unique value in that same second column. I'd like the output to be value,count sorted by most instances. Thanks for any guidance! Data example: 317476,317756,0 816063,318861,0 313123,319091,0... (4 Replies)
Discussion started by: batcho
4 Replies

3. Shell Programming and Scripting

Find and count unique date values in a file based on position

Hello, I need some sort of way to extract every date contained in a file, and count how many of those dates there are. Here are the specifics: The date format I'm looking for is mm/dd/yyyy I only need to look after line 45 in the file (that's where the data begins) The columns of... (2 Replies)
Discussion started by: ronan1219
2 Replies

4. Shell Programming and Scripting

Compare multiple files, identify common records and combine unique values into one file

Good morning all, I have a problem that is one step beyond a standard awk compare. I would like to compare three files which have several thousand records against a fourth file. All of them have a value in each row that is identical, and one value in each of those rows which may be duplicated... (1 Reply)
Discussion started by: nashton
1 Replies

5. Shell Programming and Scripting

Looping through entire directory and count unique values

Hello, I`m a complete newbie to coding, please help with this problem. I have multiple files in a directory, I have to loop through the contents of each file and extract number of unique isoforms in that file. Each file is tab delimited and only the line with the first parent (column 3)... (1 Reply)
Discussion started by: ritakadm
1 Replies

6. Shell Programming and Scripting

Count frequency of unique values in specific column

Hi, I have tab-deliminated data similar to the following: dot is-big 2 dot is-round 3 dot is-gray 4 cat is-big 3 hot in-summer 5 I want to count the frequency of each individual "unique" value in the 1st column. Thus, the desired output would be as follows: dot 3 cat 1 hot 1 is... (5 Replies)
Discussion started by: owwow14
5 Replies

7. Shell Programming and Scripting

Reading multiple values from multiple lines and columns and setting them to unique variables.

Hello, I would like to ask for help with csh script. An example of an input in .txt file is below, the number of lines varies from file to file and I have 2 or 3 columns with values. I would like to read all the values (probably one by one) and set them to independent unique variables that... (7 Replies)
Discussion started by: FMMOLA
7 Replies

8. Shell Programming and Scripting

Comparing multiple network files (edge lists)

I want to compare 4 edge-lists to basically see if an edge is present in all 4 networks. The issue is that an edge A-B in one file can be present as B-A in another file. Example: Input 1: net1.txt A B 0.1 C D 0.65 D E 0.9 E A 0.7 Input 2: net2.txt A Z 0.1 C D 0.65 E D 0.9 E A... (1 Reply)
Discussion started by: Sanchari
1 Replies

9. Shell Programming and Scripting

Print count of unique values

Hello experts, I am converting a number into its binary output as : read n echo "obase=2;$n" | bc I wish to count the maximum continuous occurrences of the digit 1. Example : 1. The binary equivalent of 5 = 101. Hence the output must be 1. 2. The binary... (3 Replies)
Discussion started by: H squared
3 Replies

10. Shell Programming and Scripting

Count number of unique values in each column of array

What is an efficient way of counting the number of unique values in a 400 column by 1000 row array and outputting the counts per column, assuming the unique values in the array are: A, B, C, D In other words the output should look like: Value COL1 COL2 COL3 A 50 51 52... (16 Replies)
Discussion started by: Geneanalyst
16 Replies

LEARN ABOUT REDHAT

dos2unix

dos2unix(1)						      General Commands Manual						       dos2unix(1)

NAME

       dos2unix - DOS/MAC to UNIX text file format converter

SYNOPSYS

       dos2unix [options] [-c convmode] [-o file ...] [-n infile outfile ...]

       Options:

       [-hkqV] [--help] [--keepdate] [--quiet] [--version]

DESCRIPTION

       This manual page documents dos2unix, the program that converts plain text files in DOS/MAC format to UNIX format.

OPTIONS

       The following options are available:

       -h --help
	      Print online help.

       -k --keepdate
	      Keep the date stamp of output file same as input file.

       -q --quiet
	      Quiet mode. Suppress all warning and messages.

       -V --version
	      Prints version information.

       -c --convmode convmode
	      Sets conversion mode. Simulates dos2unix under SunOS.

       -o --oldfile file ...
	      Old file mode. Convert the file and write output to it. The program default to run in this mode. Wildcard names may be used.

       -n --newfile infile outfile ...
	      New  file  mode.	Convert the infile and write output to outfile. File names must be given in pairs and wildcard names should NOT be
	      used or you WILL lost your files.

EXAMPLES

       Get input from stdin and write output to stdout.

	      dos2unix

       Convert and replace a.txt. Convert and replace b.txt.

	      dos2unix a.txt b.txt

	      dos2unix -o a.txt b.txt

       Convert and replace a.txt in ASCII conversion mode.  Convert and replace b.txt in ISO conversion mode.  Convert	c.txt  from  Mac  to  Unix
       ascii format.

	      dos2unix a.txt -c iso b.txt

	      dos2unix -c ascii a.txt -c iso b.txt

	      dos2unix -c mac a.txt b.txt

       Convert and replace a.txt while keeping original date stamp.

	      dos2unix -k a.txt

	      dos2unix -k -o a.txt

       Convert a.txt and write to e.txt.

	      dos2unix -n a.txt e.txt

       Convert a.txt and write to e.txt, keep date stamp of e.txt same as a.txt.

	      dos2unix -k -n a.txt e.txt

       Convert and replace a.txt. Convert b.txt and write to e.txt.

	      dos2unix a.txt -n b.txt e.txt

	      dos2unix -o a.txt -n b.txt e.txt

       Convert c.txt and write to e.txt. Convert and replace a.txt.  Convert and replace b.txt. Convert d.txt and write to f.txt.

	      dos2unix -n c.txt e.txt -o a.txt b.txt -n d.txt f.txt

DIAGNOSTICS

BUGS

       The program does not work properly under MSDOS in stdio processing mode.  If you know why is that so, please tell me.

AUTHORS

       Benjamin Lin - <blin@socs.uts.edu.au>

       Bernd Johannes Wuebben (mac2unix mode) <wuebben@kde.org>

MISCELLANY

       Tested environment:

	      Linux 1.2.0 with GNU C 2.5.8

	      SunOS 4.1.3 with GNU C 2.6.3

	      MS-DOS 6.20 with Borland C++ 4.02

       Suggestions and bug reports are welcome.

SEE ALSO

       unix2dos(1) mac2unix(1)

1995.03.31							   dos2unix v3.0						       dos2unix(1)
All times are GMT -4. The time now is 06:57 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy