08-28-2017
Are all of your .csv files sorted on field 38? If not, your code won't work. (You will get artificially high counts for the number of distinct values in a file because uniq produces a line of output for each case where a field 38 value changes from the value found on the previous line.)
Does each distinct value in field 38 of all of your files appear in only one or your input files? If not, your code won't work. (You don't have any way to determine which distinct values in a single file appear in one or more of the other files.)
Are you always processing 3 files?
Having 3 files of a megabyte each each should not cause any problem producing a single merged or sorted combined file. Why are you unable to merge them?
Why not just use a single awk script to read all of your files once and produce the output you want for each input file and for the combined input from all of the input files?
Do you really want the number of distinct field 38 values in each input file? Or, do you really just want the number of distinct field 38 values in the merged input files?
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi,
I have a challenging task,in which i have to find the duplicate files by its name and size,then i need to take anyone of the file.Then i need to open the file and find for more than one pattern and count of that pattern.
Note:These are the samples of two files,but i can have more... (2 Replies)
Discussion started by: jerome Sukumar
2 Replies
2. UNIX for Dummies Questions & Answers
Ok, another fun hiccup in my UNIX learning curve. I am trying to count the number of occurrences of an IP address across multiple files named example.hits. I can extract the number of occurrences from the files individually but when you use grep -c with multiple files you get the output similar to... (5 Replies)
Discussion started by: MrAd
5 Replies
3. Shell Programming and Scripting
Seems like can use awk and perl command. But I don't have the idea to write the command line. Thanks for all of your advise.
For example, if I have the file whose content are:
Sample 1. ATAGCAGAGGGAGTGAAGAGGTGGTGGGAGGGAGCT
Sample 2. ACTTTTATTTGAATGTAATATTTGGGACAATTATTC
Sample 3.... (1 Reply)
Discussion started by: patrick chia
1 Replies
4. Shell Programming and Scripting
how to count the total number of lines of all the files under a directory using perl script..
I mean if I have 10 files under a directory then I want to count the total number of lines of all the 10 files contain. Please help me in writing a perl script on this. (5 Replies)
Discussion started by: adityam
5 Replies
5. Shell Programming and Scripting
Please advice how can we search for a string say (abc) in multiple files and to get total occurrence of that searched string. (Need number of records that exits in period of time).
File look like this (read as filename.yyyymmdd)
a.20100101
b.20100108
c.20100115
d.20100122
e.20100129... (2 Replies)
Discussion started by: zooby
2 Replies
6. Shell Programming and Scripting
i want to find the no:of occurrences of a word in a file
cat 1.txt
unix script unix script
unix script unix script unix script unix script
unix script unix script unix
unix
script
unix script unix script now i want to find , how many times 'unix' was occurred
please help me
thanks... (6 Replies)
Discussion started by: mahesh1987
6 Replies
7. Shell Programming and Scripting
Hi Guys,
I need to write a script to compare the count of two csv files each having 5 columns.
Everyday a csv file is recived.
Now we need to compare the count of todays csv file with yesterday's csv file and if the total count of records is same in todays csv file and yesterday csv file out... (3 Replies)
Discussion started by: Vivekit82
3 Replies
8. Shell Programming and Scripting
Hi,
Very good wishes to all!
Please help to provide the shell script for generating the record counts in filed wise from the .csv file
My question:
Source file:
Field1 Field2 Field3
abc 12f sLm
1234 hjd 12d
Hyd 34
Chn
My target file should generate the .csv file with the... (14 Replies)
Discussion started by: Kirands
14 Replies
9. Shell Programming and Scripting
Hi,
I have a .dat file with contents like the below:
Input file
============SEQ NO-1: COLUMN1==========
9835619
7152815
============SEQ NO-2: COLUMN2 ==========
7615348
7015548
9373086
============SEQ NO-3: COLUMN3===========
9373086
Expected Output: (I just... (1 Reply)
Discussion started by: MS06
1 Replies
10. UNIX for Beginners Questions & Answers
Hello All,
just wanted to export multiple tables from oracle sql using unix shell script to csv file and the below code is exporting only the first table.
Can you please suggest why? or any better idea?
export FILE="/abc/autom/file/geo_JOB.csv"
Export= `sqlplus -s dev01/password@dEV3... (16 Replies)
Discussion started by: Hope
16 Replies
LEARN ABOUT XFREE86
total
TOTAL(1) General Commands Manual TOTAL(1)
NAME
total - sum up columns
SYNOPSIS
total [ -m ][ -sE | -p | -u | -l ][ -i{f|d}[N] ][ -o{f|d} ][ -tC ][ -N [ -r ]] [ file .. ]
DESCRIPTION
Total sums up columns of real numbers from one or more files and prints out the result on its standard output.
By default, total computes the straigt sum of each input column, but multiplication can be specified instead with the -p option. Likewise,
the -u option means find the upper limit (maximum), and -l means find the lower limit (minimum).
Sums of powers can be computed by giving an exponent with the -s option. (Note that there is no space between the -s and the exponent.)
This exponent can be any real number, positive or negative. The absolute value of the input is always taken before the power is computed
in order to avoid complex results. Thus, -s1 will produce a sum of absolute values. The default power (zero) is interpreted as a straight
sum without taking absolute values.
The -m option can be used to compute the mean rather than the total. For sums, the arithmetic mean is computed. For products, the geomet-
ric mean is computed. (A logarithmic sum of absolute values is used to avoid overflow, and zero values are silently ignored.)
If the input data is binary, the -id or -if option may be given for 64-bit double or 32-bit float values, respectively. Either option may
be followed immediately by an optional count, which defaults to 1, indicating the number of double or float binary values to read per
record on the input file. (There can be no space between the option and this count.) Similarly, the -od and -of options specify binary
double or float output, respectively. These options do not need a count, as this will be determined by the number of input channels.
A count can be given as the number of lines to read before computing a result. Normally, total reads each file to its end before producing
its result, but this behavior may be overridden by inserting blank lines in the input. For each blank input line, total produces a result
as if the end-of-file had been reached. If two blank lines immediately follow each other, total closes the file and proceeds to the next
one (after reporting the result). The -N option (where N is a decimal integer) tells total to produce a result and reset the calculation
after every N input lines. In addition, the -r option can be specified to override reinitialization and thus give a running total every N
lines (or every blank line). If the end of file is reached, the current total is printed and the calculation is reset before the next file
(with or without the -r option).
The -tC option can be used to specify the input and output tab character. The default tab character is TAB.
If no files are given, the standard input is read.
EXAMPLE
To compute the RMS value of colon-separated columns in a file:
total -t: -m -s2 input
To produce a running product of values from a file:
total -p -1 -r input
BUGS
If the input files have varying numbers of columns, mean values will certainly be off. Total will ignore missing column entries if the tab
separator is a non-white character, but cannot tell where a missing column should have been if the tab character is white.
AUTHOR
Greg Ward
SEE ALSO
cnt(1), neaten(1), rcalc(1), rlam(1), tabfunc(1)
RADIANCE
2/3/95 TOTAL(1)