Sponsored Content
Full Discussion: Huge files manipulation
Top Forums UNIX for Advanced & Expert Users Huge files manipulation Post 302255480 by Klashxx on Thursday 6th of November 2008 01:27:06 PM
Old 11-06-2008
Huge files manipulation

Hi , i need a fast way to delete duplicates entrys from very huge files ( >2 Gbs ) , these files are in plain text.

I tried all the usual methods ( awk / sort /uniq / sed /grep .. ) but it always ended with the same result (memory core dump)

In using HP-UX large servers.

Any advice will be very well come.


Thx in advance.

PD:I do not want to split the files.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Comparing two huge files

Hi, I have two files file A and File B. File A is a error file and File B is source file. In the error file. First line is the actual error and second line gives the information about the record (client ID) that throws error. I need to compare the first field (which doesnt start with '//') of... (11 Replies)
Discussion started by: kmkbuddy_1983
11 Replies

2. UNIX for Dummies Questions & Answers

Difference between two huge files

Hi, As per my requirement, I need to take difference between two big files(around 6.5 GB) and get the difference to a output file without any line numbers or '<' or '>' in front of each new line. As DIFF command wont work for big files, i tried to use BDIFF instead. I am getting incorrect... (13 Replies)
Discussion started by: pyaranoid
13 Replies

3. High Performance Computing

Huge Files to be Joined on Ux instead of ORACLE

we have one file (11 Million) line that is being matched with (10 Billion) line. the proof of concept we are trying , is to join them on Unix : All files are delimited and they have composite keys.. could unix be faster than Oracle in This regards.. Please advice (1 Reply)
Discussion started by: magedfawzy
1 Replies

4. Shell Programming and Scripting

Split a huge data into few different files?!

Input file data contents: >seq_1 MSNQSPPQSQRPGHSHSHSHSHAGLASSTSSHSNPSANASYNLNGPRTGGDQRYRASVDA >seq_2 AGAAGRGWGRDVTAAASPNPRNGGGRPASDLLSVGNAGGQASFASPETIDRWFEDLQHYE >seq_3 ATLEEMAAASLDANFKEELSAIEQWFRVLSEAERTAALYSLLQSSTQVQMRFFVTVLQQM ARADPITALLSPANPGQASMEAQMDAKLAAMGLKSPASPAVRQYARQSLSGDTYLSPHSA... (7 Replies)
Discussion started by: patrick87
7 Replies

5. Shell Programming and Scripting

Splitting the Huge file into several files...

Hi I have to write a script to split the huge file into several pieces. The file columns is | pipe delimited. The data sample is as: 6625060|1420215|07308806|N|20100120|5572477081|+0002.79|+0000.00|0004|0001|......... (3 Replies)
Discussion started by: lakteja
3 Replies

6. Shell Programming and Scripting

Compare 2 folders to find several missing files among huge amounts of files.

Hi, all: I've got two folders, say, "folder1" and "folder2". Under each, there are thousands of files. It's quite obvious that there are some files missing in each. I just would like to find them. I believe this can be done by "diff" command. However, if I change the above question a... (1 Reply)
Discussion started by: jiapei100
1 Replies

7. Shell Programming and Scripting

Comparing 2 huge text files

I have this 2 files: k5login sanwar@systems.nyfix.com jjamnik@systems.nyfix.com nisha@SYSTEMS.NYFIX.COM rdpena@SYSTEMS.NYFIX.COM service/backups-ora@SYSTEMS.NYFIX.COM ivanr@SYSTEMS.NYFIX.COM nasapova@SYSTEMS.NYFIX.COM tpulay@SYSTEMS.NYFIX.COM rsueno@SYSTEMS.NYFIX.COM... (11 Replies)
Discussion started by: linuxgeek
11 Replies

8. Shell Programming and Scripting

Compression - Exclude huge files

I have a DB folder which sizes to 60GB approx. It has logs which size from 500MB - 1GB. I have an Installation which would update the DB. I need to backup this DB folder, just incase my Installation FAILS. But I do not need the logs in my backup. How do I exclude them during compression (tar)? ... (2 Replies)
Discussion started by: DevendraG
2 Replies

9. UNIX for Dummies Questions & Answers

File comparison of huge files

Hi all, I hope you are well. I am very happy to see your contribution. I am eager to become part of it. I have the following question. I have two huge files to compare (almost 3GB each). The files are simulation outputs. The format of the files are as below For clear picture, please see... (9 Replies)
Discussion started by: kaaliakahn
9 Replies

10. Shell Programming and Scripting

Aggregation of Huge files

Hi Friends !! I am facing a hash total issue while performing over a set of files of huge volume: Command used: tail -n +2 <File_Name> |nawk -F"|" -v '%.2f' qq='"' '{gsub(qq,"");sa+=($156<0)?-$156:$156}END{print sa}' OFMT='%.5f' Pipe delimited file and 156 column is for hash totalling.... (14 Replies)
Discussion started by: Ravichander
14 Replies
xowish(1)							XOTcl Applications							 xowish(1)

__________________________________________________________________________________________________________________________________________________

NAME
xowish - Graphical shell containing object-oriented scripting language XOTcl SYNOPSIS
xowish ?fileName arg arg ...? _________________________________________________________________ DESCRIPTION
xowish is a shell-like application that reads XOTcl commands from its standard input or from a file and evaluates them. In addition to xot- clsh it provides graphical user interface support for TK widgets. XOTcl (XOTcl, pronounced exotickle) is an object-oriented scripting language based on MIT's OTcl. It is intended as a value added replace- ment for OTcl. Scripting languages, like Tcl, are designed for glueing components together, provide features like dynamic extensibility and dynamic typing with automatic conversion, that make them well suited for rapid application development. The basic object system of XOTcl is adopted from OTcl. The object system enables us to define objects, classes, and meta-classes. Classes are special objects with the purpose of managing other objects. ``Managing'' means that a class controls the creation and destruction of its instances and that it contains a repository of methods accessible for the instances. Every object may be enhanced with object-specific methods. XOTcl supports single and multiple inheritance. All relationships in XOTcl, including class and superclass relationships, are com- pletely dynamic and can be introspected. Through method chaining without explicit naming of the intended method, ambiguities in name reso- lution of methods are avoided. This way a shadowed method can be ``mixed into'' the execution of the current method. XOTcl combines the ideas of scripting and object-orientation in a way that preserves the benefits of both of them. It is equipped with sev- eral new language functionalities that help building and managing complex systems. We added the following support: Dynamic Object Aggregations, to provide dynamic aggregations through nested namespaces (objects). Nested Classes, to reduce the interference of independently developed program structures. Assertions, to reduce the interface and the reliability problems caused by dynamic typing and, therefore, to ease the combination of many components. Meta-data, to enhance self-documentation of objects and classes. Per-object mixins, as a means to improve flexibility of mixin methods by giving an object access to several different supplemental classes, which may be changed dynamically. Per-class mixins, as a means to improve flexibility of mixin methods to a class, all instances of the class have access to the mixed in methods like for multiple inheritance, but without the need of intersection classes. Filters as a means of abstractions over method invocations to implement large program structures, like design patterns. Dynamic Component Loading XOTcl integrates the Tcl package loading with architectrual support for integration with object-oriented con- structs. Moreover, it provides tracking/tracing of component loading. KEYWORDS
argument, interpreter, prompt, script file, shell XOWish xowish(1)
All times are GMT -4. The time now is 06:39 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy