Remove dupes in a large file Post: 303024652

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

remove a large number of user from oracle

Hi on solaris and oracle 10g2, I have number of users created in Oracle, I wonder if I have a list of the usernames will it be possible to remove the users quickly ? I want to keep the users access to system but oracle. some thing like shell script may be ?:confused: I am trying to...

2. Shell Programming and Scripting

Sed or awk script to remove text / or perform calculations from large CSV files

I have a large CSV files (e.g. 2 million records) and am hoping to do one of two things. I have been trying to use awk and sed but am a newbie and can't figure out how to get it to work. Any help you could offer would be greatly appreciated - I'm stuck trying to remove the colon and wildcards in...

3. Shell Programming and Scripting

remove a specific line in a LARGE file

Hi guys, i have a really big file, and i want to remove a specific line. sed -i '5d' fileThis doesn't really work, it takes a lot of time... The whole script is supposed to remove every word containing less than 5 characters and currently looks like this: #!/bin/bash line="1"...

4. Shell Programming and Scripting

Remove Duplicate Filenames in 2 very large directories

Hello Gurus, O/S RHEL4 I have a requirement to compare two linux based directories for duplicate filenames and remove them. These directories are close to 2 TB each. I have tried running a: Prompt>diff -r data1/ data2/ I have tried this as well: jason@jason-desktop:~$ cat script.sh ...

5. Shell Programming and Scripting

How to remove a subset of data from a large dataset based on values on one line

Hello. I was wondering if anyone could help. I have a file containing a large table in the format: marker1 marker2 marker3 marker4 position1 position2 position3 position4 genotype1 genotype2 genotype3 genotype4 with marker being a name, position a numeric...

6. UNIX for Dummies Questions & Answers

Filtering F-Dupes

Is there an easy way to tell FDupes what filetypes to look at or ignore?

7. Shell Programming and Scripting

Removing Dupes from huge file- awk/perl/uniq

Hi, I have the following command in place nawk -F, '!a++' file > file.uniq It has been working perfectly as per requirements, by removing duplicates by taking into consideration only first 3 fields. Recently it has started giving below error: bash-3.2$ nawk -F, '!a++'...

8. Shell Programming and Scripting

remove large portion of web page code between two tags

Hi everybody, I am trying to remove bunch of lines from web pages between two tags: one is <h1> and the other is <table it looks like <h1>Anniversary cards roses</h1> many lines here <table summary="Free anniversary greeting cards." cellspacing="8" cellpadding="8" width="70%">my goal...

9. Shell Programming and Scripting

Removing dupes within 2 delimited areas in a large dictionary file

Hello, I have a very large dictionary file which is in text format and which contains a large number of sub-sections. Each sub-section starts with the following header : #DATA #VALID 1 and ends with a footer as shown below #END The data between the Header and the Footer consists of...

10. Shell Programming and Scripting

Modify script to remove dupes with two delimiters

Hello, I have a script which removes duplicates in a database with a single delimiter = The script is given below: # script to remove dupes from a row with structure word=word BEGIN{FS="="} {for(i=1;i<=NF;i++){a++;}for(i in a){b=b"="i}{sub("=","",b);$0=b;b="";delete a}}1 How do I modify...

LEARN ABOUT DEBIAN

type_numeric

form_variables(3FORM)													     form_variables(3FORM)

NAME

       TYPE_ALNUM, TYPE_ALPHA, TYPE_ENUM, TYPE_INTEGER, TYPE_IPV4, TYPE_NUMERIC, TYPE_REGEXP - form system global variables

SYNOPSIS

       #include <form.h>

       FIELDTYPE * TYPE_ALNUM;
       FIELDTYPE * TYPE_ALPHA;
       FIELDTYPE * TYPE_ENUM;
       FIELDTYPE * TYPE_INTEGER;
       FIELDTYPE * TYPE_IPV4;
       FIELDTYPE * TYPE_NUMERIC;
       FIELDTYPE * TYPE_REGEXP;

DESCRIPTION

       These  are  building blocks for the form library, defining fields that can be created using set_fieldtype(3X).  Each provides functions for
       field- and character-validation, according to the given datatype.

   TYPE_ALNUM
       This holds alphanumeric data.

   TYPE_ALPHA
       This holds alphabetic data.

   TYPE_ENUM
       This holds an enumerated type.

   TYPE_INTEGER
       This holds a decimal integer.

   TYPE_IPV4
       This holds an IPv4 internet address, e.g., "127.0.0.1".

   TYPE_NUMERIC
       This holds a decimal number, with optional sign and decimal point.

   TYPE_REGEXP
       This holds a regular expression.

PORTABILITY

       The TYPE_IPV4 variable is an extension not provided by older implementations of the form library.

SEE ALSO

       form(3FORM).

															     form_variables(3FORM)