Finding total distinct count from multiple csv files through UNIX script
Hi All ,
I have multiple pipe delimited csv files are present in a directory.I need to find out distinct count on a column on those files and need the total distinct
count on all files.
We can't merge all the files here as file size are huge in millions.I have tried in below way for each files.
I need to automate the above procedure (ex: by putting each files count in a temporary file)as multiple csv files are present in the directory as I need the total distinct count on a column on all those files.Can anyone please help me with this.
Last edited by Don Cragun; 08-28-2017 at 04:00 AM..
Reason: Change HTML tags to CODE tags.
Hi,
I have a challenging task,in which i have to find the duplicate files by its name and size,then i need to take anyone of the file.Then i need to open the file and find for more than one pattern and count of that pattern.
Note:These are the samples of two files,but i can have more... (2 Replies)
Ok, another fun hiccup in my UNIX learning curve. I am trying to count the number of occurrences of an IP address across multiple files named example.hits. I can extract the number of occurrences from the files individually but when you use grep -c with multiple files you get the output similar to... (5 Replies)
Seems like can use awk and perl command. But I don't have the idea to write the command line. Thanks for all of your advise.
For example, if I have the file whose content are:
Sample 1. ATAGCAGAGGGAGTGAAGAGGTGGTGGGAGGGAGCT
Sample 2. ACTTTTATTTGAATGTAATATTTGGGACAATTATTC
Sample 3.... (1 Reply)
how to count the total number of lines of all the files under a directory using perl script..
I mean if I have 10 files under a directory then I want to count the total number of lines of all the 10 files contain. Please help me in writing a perl script on this. (5 Replies)
Please advice how can we search for a string say (abc) in multiple files and to get total occurrence of that searched string. (Need number of records that exits in period of time).
File look like this (read as filename.yyyymmdd)
a.20100101
b.20100108
c.20100115
d.20100122
e.20100129... (2 Replies)
i want to find the no:of occurrences of a word in a file
cat 1.txt
unix script unix script
unix script unix script unix script unix script
unix script unix script unix
unix
script
unix script unix script now i want to find , how many times 'unix' was occurred
please help me
thanks... (6 Replies)
Hi Guys,
I need to write a script to compare the count of two csv files each having 5 columns.
Everyday a csv file is recived.
Now we need to compare the count of todays csv file with yesterday's csv file and if the total count of records is same in todays csv file and yesterday csv file out... (3 Replies)
Hi,
Very good wishes to all!
Please help to provide the shell script for generating the record counts in filed wise from the .csv file
My question:
Source file:
Field1 Field2 Field3
abc 12f sLm
1234 hjd 12d
Hyd 34
Chn
My target file should generate the .csv file with the... (14 Replies)
Hi,
I have a .dat file with contents like the below:
Input file
============SEQ NO-1: COLUMN1==========
9835619
7152815
============SEQ NO-2: COLUMN2 ==========
7615348
7015548
9373086
============SEQ NO-3: COLUMN3===========
9373086
Expected Output: (I just... (1 Reply)
Hello All,
just wanted to export multiple tables from oracle sql using unix shell script to csv file and the below code is exporting only the first table.
Can you please suggest why? or any better idea?
export FILE="/abc/autom/file/geo_JOB.csv"
Export= `sqlplus -s dev01/password@dEV3... (16 Replies)
Discussion started by: Hope
16 Replies
LEARN ABOUT PHP
total
TOTAL(1) General Commands Manual TOTAL(1)NAME
total - sum up columns
SYNOPSIS
total [ -m ][ -sE | -p | -u | -l ][ -i{f|d}[N] ][ -o{f|d} ][ -tC ][ -N [ -r ]] [ file .. ]
DESCRIPTION
Total sums up columns of real numbers from one or more files and prints out the result on its standard output.
By default, total computes the straigt sum of each input column, but multiplication can be specified instead with the -p option. Likewise,
the -u option means find the upper limit (maximum), and -l means find the lower limit (minimum).
Sums of powers can be computed by giving an exponent with the -s option. (Note that there is no space between the -s and the exponent.)
This exponent can be any real number, positive or negative. The absolute value of the input is always taken before the power is computed
in order to avoid complex results. Thus, -s1 will produce a sum of absolute values. The default power (zero) is interpreted as a straight
sum without taking absolute values.
The -m option can be used to compute the mean rather than the total. For sums, the arithmetic mean is computed. For products, the geomet-
ric mean is computed. (A logarithmic sum of absolute values is used to avoid overflow, and zero values are silently ignored.)
If the input data is binary, the -id or -if option may be given for 64-bit double or 32-bit float values, respectively. Either option may
be followed immediately by an optional count, which defaults to 1, indicating the number of double or float binary values to read per
record on the input file. (There can be no space between the option and this count.) Similarly, the -od and -of options specify binary
double or float output, respectively. These options do not need a count, as this will be determined by the number of input channels.
A count can be given as the number of lines to read before computing a result. Normally, total reads each file to its end before producing
its result, but this behavior may be overridden by inserting blank lines in the input. For each blank input line, total produces a result
as if the end-of-file had been reached. If two blank lines immediately follow each other, total closes the file and proceeds to the next
one (after reporting the result). The -N option (where N is a decimal integer) tells total to produce a result and reset the calculation
after every N input lines. In addition, the -r option can be specified to override reinitialization and thus give a running total every N
lines (or every blank line). If the end of file is reached, the current total is printed and the calculation is reset before the next file
(with or without the -r option).
The -tC option can be used to specify the input and output tab character. The default tab character is TAB.
If no files are given, the standard input is read.
EXAMPLE
To compute the RMS value of colon-separated columns in a file:
total -t: -m -s2 input
To produce a running product of values from a file:
total -p -1 -r input
BUGS
If the input files have varying numbers of columns, mean values will certainly be off. Total will ignore missing column entries if the tab
separator is a non-white character, but cannot tell where a missing column should have been if the tab character is white.
AUTHOR
Greg Ward
SEE ALSO cnt(1), neaten(1), rcalc(1), rlam(1), tabfunc(1)RADIANCE 2/3/95 TOTAL(1)