10-25-2018
File comaprsons for the Huge data files ( around 60G) - Need optimized and teh best way to do this
I have 2 large file (.dat) around 70 g, 12 columns but the data not sorted in both the files.. need your inputs in giving the best optimized method/command to achieve this and redirect the not macthing lines to the thrid file ( diff.dat)
File 1 - 15 columns
File 2 - 15 columns
Data is not in sorted order.
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
folks,
In my working directory, there a multiple large files which only contain one line in the file. The line is too long to use "grep", so any help?
For example, if I want to find if these files contain a string like "93849", what command I should use?
Also, there is oder_id number... (1 Reply)
Discussion started by: ting123
1 Replies
2. Shell Programming and Scripting
Hi,
I have a huge file of bibliographic records in some standard format.I need a script to do some repeatable task as follows:
1. Needs to create folders as the strings starts with "item_*" from the input file
2. Create a file "contents" in each folders having "license.txt(tab... (5 Replies)
Discussion started by: srsahu75
5 Replies
3. Shell Programming and Scripting
I have a file with data extracted, and need to insert a header with a constant string, say: H|PayerDataExtract
if i use sed, i have to redirect the output to a seperate file like
sed ' sed commands' ExtractDataFile.dat > ExtractDataFileWithHeader.dat
the same is true for awk
and... (10 Replies)
Discussion started by: deepaktanna
10 Replies
4. Shell Programming and Scripting
Input file data contents:
>seq_1
MSNQSPPQSQRPGHSHSHSHSHAGLASSTSSHSNPSANASYNLNGPRTGGDQRYRASVDA
>seq_2
AGAAGRGWGRDVTAAASPNPRNGGGRPASDLLSVGNAGGQASFASPETIDRWFEDLQHYE
>seq_3
ATLEEMAAASLDANFKEELSAIEQWFRVLSEAERTAALYSLLQSSTQVQMRFFVTVLQQM
ARADPITALLSPANPGQASMEAQMDAKLAAMGLKSPASPAVRQYARQSLSGDTYLSPHSA... (7 Replies)
Discussion started by: patrick87
7 Replies
5. Shell Programming and Scripting
Hi
I have to write a script to split the huge file into several pieces. The file columns is | pipe delimited. The data sample is as:
6625060|1420215|07308806|N|20100120|5572477081|+0002.79|+0000.00|0004|0001|......... (3 Replies)
Discussion started by: lakteja
3 Replies
6. Shell Programming and Scripting
Hello Everyone,
I have a perl script that reads two types of data files (txt and XML). These data files are huge and large in number. I am using something like this :
foreach my $t (@text)
{
open TEXT, $t or die "Cannot open $t for reading: $!\n";
while(my $line=<TEXT>){
... (4 Replies)
Discussion started by: ad23
4 Replies
7. Shell Programming and Scripting
I got three different file:
Part of File 1
ARTPHDFGAA
.
.
Part of File 2
ARTGHHYESA
.
.
Part of File 3
ARTPOLYWEA
.
. (4 Replies)
Discussion started by: patrick87
4 Replies
8. Shell Programming and Scripting
I’m new to Linux script and not sure how to filter out bad records from huge flat files (over 1.3GB each). The delimiter is a semi colon “;”
Here is the sample of 5 lines in the file:
Name1;phone1;address1;city1;state1;zipcode1
Name2;phone2;address2;city2;state2;zipcode2;comment... (7 Replies)
Discussion started by: lv99
7 Replies
9. UNIX for Dummies Questions & Answers
Hi all,
I hope you are well. I am very happy to see your contribution. I am eager to become part of it.
I have the following question. I have two huge files to compare (almost 3GB each). The files are simulation outputs. The format of the files are as below
For clear picture, please see... (9 Replies)
Discussion started by: kaaliakahn
9 Replies
10. UNIX for Advanced & Expert Users
Optimization shell/awk script to aggregate (sum) for all the columns of Huge data file
File delimiter "|"
Need to have Sum of all columns, with column number : aggregation (summation) for each column
File not having the header
Like below -
Column 1 "Total
Column 2 : "Total
...
...... (2 Replies)
Discussion started by: kartikirans
2 Replies
LEARN ABOUT XFREE86
histo
HISTO(1) General Commands Manual HISTO(1)
NAME
histo - compute 1-dimensional histogram of N data columns
SYNOPSIS
histo [-c][-p] xmin xmax nbins
histo [-c][-p] imin imax
DESCRIPTION
Histo bins columnular data on the standard input between the given minimum and maximum values. If three command line arguments are given,
the third is taken as the number of data bins between the first two real numbers. If only two arguments are given, they are both assumed
to be integers, and the number of data bins will be equal to their difference plus one. The bins are always of equal size.
The output is N+1 columns of data (for N columns input), where the first column is the centroid of each division, and each row corresponds
to the frequencies for each column around that value.
If the -c option is present, then histo computes the cumulative histogram for each column instead of the straight frequencies. The upper
value of each bin is printed also instead of the centroid. This may be useful in computing percentiles, for example. Values below the
minimum specified are still counted in the cumulative total.
The -p option tells histo to report the percentage of the total number of input lines rather than the absolute counts. In the case of a
cumulative total, this yields the percentile values directly. Values above the maximum are counted as well as values below in this case.
All input data is interpreted as real values, and columns must be white-space separated. If any value is less than the minimum or greater
than the maximum, it will be ignored unless the -c option is specified.
EXAMPLE
To count data values between -1 and 1 in 50 bins:
histo -1 1 50 < input.dat
To count frequencies of integers between 0 and 255:
histo 0 255 < input.dat
AUTHOR
Greg Ward
SEE ALSO
cnt(1), neaten(1), rcalc(1), rlam(1), tabfunc(1), total(1)
RADIANCE
9/6/96 HISTO(1)