Sponsored Content
Full Discussion: Huge files manipulation
Top Forums UNIX for Advanced & Expert Users Huge files manipulation Post 302255505 by jim mcnamara on Thursday 6th of November 2008 02:33:43 PM
Old 11-06-2008
If your key first character is not highly redundant you can try this with awk.
The idea is predicated on your original code blowing the limits for a hash:
Code:
#example with a numeric value as the first char of key
# this uses a concatenated key

awk '{ key=substr($0,10,3) substr($0,35,10)
       ch=substr(key,1,1)
       if(ch="0") if(!arr0[key]++) {print $0; continue}
       if(ch="1") if(!arr1[key]++) {print $0; continue}
       if(ch="2") if(!arr2[key]++) {print $0; continue}
       if(ch="3") if(!arr3[key]++) {print $0; continue}     
       if(ch="4") if(!arr4[key]++) {print $0; continue}
       if(ch="5") if(!arr5[key]++) {print $0; continue}
       if(ch="6") if(!arr6[key]++) {print $0; continue}
       if(ch="7") if(!arr7[key]++) {print $0; continue}
       if(ch="8") if(!arr8[key]++) {print $0; continue}
       if(ch="9") if(!arr9[key]++) {print $0;}
      }'  inputfile  > outputfile

This worked for me with a > 2GB file on a V class 9000 11.0
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Comparing two huge files

Hi, I have two files file A and File B. File A is a error file and File B is source file. In the error file. First line is the actual error and second line gives the information about the record (client ID) that throws error. I need to compare the first field (which doesnt start with '//') of... (11 Replies)
Discussion started by: kmkbuddy_1983
11 Replies

2. UNIX for Dummies Questions & Answers

Difference between two huge files

Hi, As per my requirement, I need to take difference between two big files(around 6.5 GB) and get the difference to a output file without any line numbers or '<' or '>' in front of each new line. As DIFF command wont work for big files, i tried to use BDIFF instead. I am getting incorrect... (13 Replies)
Discussion started by: pyaranoid
13 Replies

3. High Performance Computing

Huge Files to be Joined on Ux instead of ORACLE

we have one file (11 Million) line that is being matched with (10 Billion) line. the proof of concept we are trying , is to join them on Unix : All files are delimited and they have composite keys.. could unix be faster than Oracle in This regards.. Please advice (1 Reply)
Discussion started by: magedfawzy
1 Replies

4. Shell Programming and Scripting

Split a huge data into few different files?!

Input file data contents: >seq_1 MSNQSPPQSQRPGHSHSHSHSHAGLASSTSSHSNPSANASYNLNGPRTGGDQRYRASVDA >seq_2 AGAAGRGWGRDVTAAASPNPRNGGGRPASDLLSVGNAGGQASFASPETIDRWFEDLQHYE >seq_3 ATLEEMAAASLDANFKEELSAIEQWFRVLSEAERTAALYSLLQSSTQVQMRFFVTVLQQM ARADPITALLSPANPGQASMEAQMDAKLAAMGLKSPASPAVRQYARQSLSGDTYLSPHSA... (7 Replies)
Discussion started by: patrick87
7 Replies

5. Shell Programming and Scripting

Splitting the Huge file into several files...

Hi I have to write a script to split the huge file into several pieces. The file columns is | pipe delimited. The data sample is as: 6625060|1420215|07308806|N|20100120|5572477081|+0002.79|+0000.00|0004|0001|......... (3 Replies)
Discussion started by: lakteja
3 Replies

6. Shell Programming and Scripting

Compare 2 folders to find several missing files among huge amounts of files.

Hi, all: I've got two folders, say, "folder1" and "folder2". Under each, there are thousands of files. It's quite obvious that there are some files missing in each. I just would like to find them. I believe this can be done by "diff" command. However, if I change the above question a... (1 Reply)
Discussion started by: jiapei100
1 Replies

7. Shell Programming and Scripting

Comparing 2 huge text files

I have this 2 files: k5login sanwar@systems.nyfix.com jjamnik@systems.nyfix.com nisha@SYSTEMS.NYFIX.COM rdpena@SYSTEMS.NYFIX.COM service/backups-ora@SYSTEMS.NYFIX.COM ivanr@SYSTEMS.NYFIX.COM nasapova@SYSTEMS.NYFIX.COM tpulay@SYSTEMS.NYFIX.COM rsueno@SYSTEMS.NYFIX.COM... (11 Replies)
Discussion started by: linuxgeek
11 Replies

8. Shell Programming and Scripting

Compression - Exclude huge files

I have a DB folder which sizes to 60GB approx. It has logs which size from 500MB - 1GB. I have an Installation which would update the DB. I need to backup this DB folder, just incase my Installation FAILS. But I do not need the logs in my backup. How do I exclude them during compression (tar)? ... (2 Replies)
Discussion started by: DevendraG
2 Replies

9. UNIX for Dummies Questions & Answers

File comparison of huge files

Hi all, I hope you are well. I am very happy to see your contribution. I am eager to become part of it. I have the following question. I have two huge files to compare (almost 3GB each). The files are simulation outputs. The format of the files are as below For clear picture, please see... (9 Replies)
Discussion started by: kaaliakahn
9 Replies

10. Shell Programming and Scripting

Aggregation of Huge files

Hi Friends !! I am facing a hash total issue while performing over a set of files of huge volume: Command used: tail -n +2 <File_Name> |nawk -F"|" -v '%.2f' qq='"' '{gsub(qq,"");sa+=($156<0)?-$156:$156}END{print sa}' OFMT='%.5f' Pipe delimited file and 156 column is for hash totalling.... (14 Replies)
Discussion started by: Ravichander
14 Replies
LIQUIDSOAP(1)						      General Commands Manual						     LIQUIDSOAP(1)

NAME
liquidsoap - a multimedia streaming language SYNOPSIS
liquidsoap [options] [script|expression] DESCRIPTION
Liquidsoap is a programming language for describing multimedia streaming systems. It is very flexible, making simple things simple but giving a lot of control for advanced uses. Liquidsoap supports audio, video and MIDI streams, and a wide range of input/output operators including Icecast and various soundcard APIs. It can perform a broad range of signal processing, combine streams in various ways, support custom transitions, generate sound procedurally... and all this can be assembled as you wish. Input files can be accessed remotely, or even be synthesized on the fly using external scripts such as speech synthesis. Finally, interaction with a running liquidsoap instance is possible via telnet or socket. Liquidsoap scripts passed on the command line will be evaluated: they shall be used to define the streaming system to be ran. It is possi- ble to pass multiple scripts; they will all be ran successively, and definitions from one script can be used in subsequent ones. A script will be read from standard input if - is given as script filename. Information about scripting liquidsoap is available on our website: http://savonet.sf.net/ If the parameter is not a file it will be treated as an expression which will be executed. It is a convenient way to test simple one-line scripts. When running only one-liners, the default is to log messages directly on stdout rather than to a file. OPTIONS
- Read script from standard input. -- Stop parsing the command-line and pass subsequent items to the script. --debug Print debugging log messages. --dynamic-plugins-dir path Directory where to look for plugins. --errors-as-warnings Issue warnings instead of fatal errors for unused variables and ignored expressions. If you are not sure about it, it is better to not use it. --interactive Start an interactive interpreter. --list-plugins List all plugins (builtin scripting values, supported formats and protocols). --list-plugins-xml List all plugins (builtin scripting values, supported formats and protocols), output as XML. --no-pervasives Do not load pervasive script libraries. --version Display liquidsoap's version. -c, --check Check and evaluate scripts but do not perform any streaming. -cl, --check-lib Like --check but treats all scripts and expressions as libraries, so that unused toplevel variables are not reported. -d, --daemon Run in daemon mode. -f, --force-start For advanced dynamic uses: force liquidsoap to start even when no active source is initially defined. -h plugin Print the description of a plugin, eg. a builtin scripting function. -i Display infered types. -p, --parse-only Parse scripts but do not type-check and run them. -q, --quiet Do not print log messages on standard output. -r filename Process a request. -T, --disable-telnet Disable the telnet server. -U, --disable-unix-socket Disable the unix socket. -t, --enable-telnet Enable the telnet server. -u, --enable-unix-socket Enable the unix socket. -v, --verbose Print log messages on standard output. --conf-descr-key key Describe a configuration key. --conf-descr Show all configuration keys with their documentation. --conf-descr-liqi Show all configuration keys with their documentation in liqi (documentation wiki) format. --conf-dump Dump the configuration state -help, --help Display this list of options SEE ALSO
Our website http://savonet.sf.net/ and the HTML documentation coming with your distribution of liquidsoap. AUTHOR
Savonet team <savonet-users@lists.sourceforge.net> Liquidsoap 1.0.1 June 25, 2012 LIQUIDSOAP(1)
All times are GMT -4. The time now is 07:06 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy