Sponsored Content
Top Forums Shell Programming and Scripting find numeric duplicates from 300 million lines.... Post 302669683 by methyl on Wednesday 11th of July 2012 08:06:39 AM
Old 07-11-2012
Is the data in any particular order? Did it come from a database?

Do you have a database engine? Some processes are just not suitable for Shell tools.
 

8 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

duplicates lines with one column different

Hi I have the following lines in a file SANDI108085FRANKLIN WRAP 7285 SANDI109514ZIPLOC STRETCH N SEAL 7285 SANDI110198CHOICE DM 0911 SANDI111144RANDOM WEIGHT BRAND 0704 SANDI111144RANDOM WEIGHT BRAND 0738... (10 Replies)
Discussion started by: dhanamurthy
10 Replies

2. Shell Programming and Scripting

How to delete lines in a file that have duplicates or derive the lines that aper once

Input: a b b c d d I need: a c I know how to get this (the lines that have duplicates) : b d sort file | uniq -d But i need opossite of this. I have searched the forum and other places as well, but have found solution for everything except this variant of the problem. (3 Replies)
Discussion started by: necroman08
3 Replies

3. Shell Programming and Scripting

Tail 86000 lines from 1.2 million line file?

I have a log file that is about 1.2 million lines long and about 300MB. we need a way to clean up this file and only keep the last few thousand lines. if i use tail command we run our of memory as the file is too big. I do have a key word to match on. example, we want to keep every line... (8 Replies)
Discussion started by: robsonde
8 Replies

4. UNIX for Dummies Questions & Answers

Find and Replace random numeric value with non-numeric value

Can someone tell me how to change the first column in a very large 17k line file from a random 10 digit numeric value to a non numeric value. The format of lines in the file is: 1702938475,SNU022,201004 the first 10 numbers always begin with 170 (6 Replies)
Discussion started by: Bahf1s
6 Replies

5. UNIX for Dummies Questions & Answers

Only print lines with 3 numeric values

Hey guys & gals, I am hoping for some advice on a sed or awk command that will allow to only print lines from a file that contain 3 numeric values. From previous searches here I saw that ygemici used the sed command to remove lines containing more than 3 numeric values ; however how... (3 Replies)
Discussion started by: TAPE
3 Replies

6. UNIX for Dummies Questions & Answers

Help with changing header of tsv with 30 million lines

Hi My 30 million line file has a header chr start end strand ref_context repeat_masked s1_smpl_context s1_c_count s1_ct_count s1_non_ct_count s1_m% s1_score s1_snp s1_indels s2_smpl_context s2_c_count s2_ct_count s2_non_ct_count s2_m% s2_score s2_snp s2_indels ... (2 Replies)
Discussion started by: plumb_r
2 Replies

7. Shell Programming and Scripting

Find duplicates in column 1 and merge their lines (awk?)

Hi, I have a file (sorted by sort) with 8 tab delimited columns. The first column contains duplicated fields and I need to merge all these identical lines. My input file: comp100002 aaa bbb ccc ddd eee fff ggg comp100003 aba aba aba aba aba aba aba comp100003 fff fff fff fff fff fff fff... (5 Replies)
Discussion started by: falcox
5 Replies

8. Shell Programming and Scripting

Fast processing(mv command) of 1 million+ files using find, mv and xargs

Hi, I'd like to ask if anybody can help improve my code to move 1 million+ files from a directory to another: find /source/dir -name file* -type f | xargs -I '{}' mv {} /destination/dir I learned this line of code from this forum as well and it works fine. However, file movement is kinda... (6 Replies)
Discussion started by: agentgrecko
6 Replies
TDB_FIRSTKEY(3) 					     Linux Programmer's Manual						   TDB_FIRSTKEY(3)

NAME
tdb_firstkey - return the key of the first record in a tdb database tdb_nextkey - return the key of next record in the tdb database SYNOPSIS
#include <tdb.h> TDB_DATA tdb_firstkey(TDB_CONTEXT *tdb); TDB_DATA tdb_nextkey(TDB_CONTEXT *tdb, TDB_DATA prev_key); DESCRIPTION
tdb_firstkey fetches the key of the first entry in the tdb database. tdb_nextkey fetches the key of the entry immediately after prev_key in the database. The TDB_DATA structure returned by both these functions is defined as: typedef struct { char *dptr; size_t dsize; } TDB_DATA; The order in which this traverses the database is its own internal hash order. Do not expect the keys to be in any recognizably sorted order. These functions are reliable even if other writers are accessing the database (or you are replacing or deleting elements as you traverse), except in the case of nested tdb_firstkey or tdb_nextkey calls. For example, the following outer traversal may fail to traverse all the elements, or may even traverse elements twice if other processes are manipulating the database: TDB_DATA i, j; for (i = tdb_firstkey(tdb); i.dptr; i = tdb_nextkey(tdb, i)) { for (j = tdb_firstkey(tdb); j.dptr; j = tdb_nextkey(tdb, j)) { ... } } If such behaviour is desired, use tdb_traverse instead. RETURN VALUE
If the call succeeds, then a TDB_DATA structure is returned with the dptr structure filled in. If the call fails or you have reached the end of the database then dptr will be set to NULL. NOTE: The caller is responsible for freeing the data pointed to by dptr AUTHORS
Software: Andrew Tridgell <tridge@linuxcare.com> and Luke Kenneth Casson Leighton Man page: Ben Woodard <ben@valinux.com> SEE ALSO
gdbm(3), tdb(3) Samba Aug 16, 2000 TDB_FIRSTKEY(3)
All times are GMT -4. The time now is 05:42 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy