Sponsored Content
Full Discussion: Aggregation of Huge files
Top Forums Shell Programming and Scripting Aggregation of Huge files Post 302891587 by Don Cragun on Thursday 6th of March 2014 02:46:41 PM
Old 03-06-2014
Making the assumption that that error code is coming from bc, you could try:
Code:
awk -F'|' -v dqANDms='["-]' '
BEGIN { f=156
        printf("s=0\n")
}
NR > 2 {gsub(dqANDms, "", $f)
        printf("s+=%s\n",  $f)
}
END {   printf("s\n")
}' file | bc

This User Gave Thanks to Don Cragun For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Comparing two huge files

Hi, I have two files file A and File B. File A is a error file and File B is source file. In the error file. First line is the actual error and second line gives the information about the record (client ID) that throws error. I need to compare the first field (which doesnt start with '//') of... (11 Replies)
Discussion started by: kmkbuddy_1983
11 Replies

2. UNIX for Dummies Questions & Answers

Difference between two huge files

Hi, As per my requirement, I need to take difference between two big files(around 6.5 GB) and get the difference to a output file without any line numbers or '<' or '>' in front of each new line. As DIFF command wont work for big files, i tried to use BDIFF instead. I am getting incorrect... (13 Replies)
Discussion started by: pyaranoid
13 Replies

3. UNIX for Advanced & Expert Users

Huge files manipulation

Hi , i need a fast way to delete duplicates entrys from very huge files ( >2 Gbs ) , these files are in plain text. I tried all the usual methods ( awk / sort /uniq / sed /grep .. ) but it always ended with the same result (memory core dump) In using HP-UX large servers. Any advice will... (8 Replies)
Discussion started by: Klashxx
8 Replies

4. High Performance Computing

Huge Files to be Joined on Ux instead of ORACLE

we have one file (11 Million) line that is being matched with (10 Billion) line. the proof of concept we are trying , is to join them on Unix : All files are delimited and they have composite keys.. could unix be faster than Oracle in This regards.. Please advice (1 Reply)
Discussion started by: magedfawzy
1 Replies

5. Shell Programming and Scripting

Help in locating a word in huge files

hi i receive about 5000 files per day in my system. Each of them are like: cat ABC.april24.dat ABH00001990 01993 409009092 0909 INI iop 9033 AAB0000237893784 8430900 898383 AUS 34349089008 849843 9474822 AAA00003849893498098394 84834 348348439 -438939 IN AAA00004438493893849384... (2 Replies)
Discussion started by: Prateek007
2 Replies

6. Shell Programming and Scripting

Compare 2 folders to find several missing files among huge amounts of files.

Hi, all: I've got two folders, say, "folder1" and "folder2". Under each, there are thousands of files. It's quite obvious that there are some files missing in each. I just would like to find them. I believe this can be done by "diff" command. However, if I change the above question a... (1 Reply)
Discussion started by: jiapei100
1 Replies

7. AIX

Copy huge files system

Dear Guy’s By using dd command or any strong command, I’d like to copy huge data from file system to another file system Sours File system: /sfsapp File system has 250 GB of data Target File system: /tgtapp I’d like to copy all these files and directories from /sfsapp to /tgtapp as... (28 Replies)
Discussion started by: Mr.AIX
28 Replies

8. Shell Programming and Scripting

Compression - Exclude huge files

I have a DB folder which sizes to 60GB approx. It has logs which size from 500MB - 1GB. I have an Installation which would update the DB. I need to backup this DB folder, just incase my Installation FAILS. But I do not need the logs in my backup. How do I exclude them during compression (tar)? ... (2 Replies)
Discussion started by: DevendraG
2 Replies

9. UNIX for Dummies Questions & Answers

File comparison of huge files

Hi all, I hope you are well. I am very happy to see your contribution. I am eager to become part of it. I have the following question. I have two huge files to compare (almost 3GB each). The files are simulation outputs. The format of the files are as below For clear picture, please see... (9 Replies)
Discussion started by: kaaliakahn
9 Replies

10. Shell Programming and Scripting

Aggregation of huge data

Hi Friends, I have a file with sample amount data as follows: -89990.3456 8788798.990000128 55109787.20 -12455558989.90876 I need to exclude the '-' symbol in order to treat all values as an absolute one and then I need to sum up.The record count is around 1 million. How... (8 Replies)
Discussion started by: Ravichander
8 Replies
FS_DISKFREE(1)						       AFS Command Reference						    FS_DISKFREE(1)

NAME
       fs_diskfree - Shows data about the partition housing a directory or file

SYNOPSIS
       fs diskfree [-path <dir/file path>+] [-human] [-help]

       fs df [-p <dir/file path>+] [-hu] [-he]

       fs di [-p <dir/file path>+] [-hu] [-he]

DESCRIPTION
       The fs diskfree command formats and displays information about the partition that houses the volume containing the specified directory or
       file, including its size and how much space is currently used.

       To display information about the volume itself, use the fs examine command. The fs examine and fs quota commands also display information
       about a volume.

CAUTIONS
       The partition-related statistics in this command's output do not always agree with the corresponding values in the output of the standard
       UNIX df command. The statistics reported by this command can be up to five minutes old, because the Cache Manager polls the File Server for
       partition information at that frequency. Also, on some operating systems, the df command's report of partition size includes reserved space
       not included in this command's calculation, and so is likely to be about 10% larger.

OPTIONS
       -path <dir/file path>+
	   Names a file or directory that resides on the partition about which to produce output. Partial pathnames are interpreted relative to
	   the current working directory, which is also the default value if this argument is omitted.

       -human
	   Print space in a "human-readable" format. Instead of always printing space in kilobytes, show disk space in kilobytes, megabytes,
	   gigabytes, terabytes, or petabytes, as appropriate.

       -help
	   Prints the online help for this command. All other valid options are ignored.

OUTPUT
       The output reports the following information about the volume and partition that houses each file or directory:

       Volume Name
	   The name of the volume.

       total
	   The partition's total size. If -human is not specified, this value is in kilobytes.

       used
	   The amount of space used on the partition. If -human is not specified, this value is in kilobytes.

       avail
	   The amount of space available on the partition. If -human is not specified, this value is in kilobytes.

       %used
	   The percentage of the partition's total space that is used (the "used" statistic divided by the "kbytes" statistic, times 100).

       If the %used statistic is greater than 90%, it is marked with the string "<<WARNING" at the right margin.

       If the volume is a read-only volume, the output includes information about only one of the partitions that houses it, generally the one on
       the file server machine with the lowest preference rank. To verify which machine the output is referring to, use the vos listvldb command
       to list the volume's locations, and the vos partinfo command to display the size of each one.

EXAMPLES
       The following example shows the output for the partitions housing the volumes "user.smith" and "sun4x_56.bin":

	  % fs diskfree -path /afs/abc.com/usr/smith /afs/abc.com/sun4x_56/bin
	  Volume Name	   total    used   avail     %used
	  user.smith	 4177920 3841258  336662       92% <<WARNING
	  sun4x_56.bin	 4423680 3174500 1249180       72%

PRIVILEGE REQUIRED
       The issuer must have the "r" (read) permission on the ACL of the root directory of the volume that houses the file or directory named by
       the -path argument, and "l" (list) permission on the ACL of each directory that precedes it in the pathname.

SEE ALSO
       fs_examine(1)

COPYRIGHT
       IBM Corporation 2000. <http://www.ibm.com/> All Rights Reserved.

       This documentation is covered by the IBM Public License Version 1.0.  It was converted from HTML to POD by software written by Chas
       Williams and Russ Allbery, based on work by Alf Wachsmann and Elizabeth Cassell.

OpenAFS 							    2012-03-26							    FS_DISKFREE(1)
All times are GMT -4. The time now is 08:26 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy