Sponsored Content
Top Forums Shell Programming and Scripting How to remove duplicates from the .dat file Post 302530727 by rdcwayx on Wednesday 15th of June 2011 12:10:29 AM
Old 06-15-2011
Similar questions are asked every week. Look for solution in forum first should be better way.

Code:
awk -F \| '!a[$1]++' infile

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicates from File from specific location

How can i remove the duplicate lines from a file, for example sample123456Sample testing123456testing XXXXX131323XXXXX YYYYY423432YYYYY fsdfdsf123456gsdfdsd all the duplicates from column 6-12 , must be deleted. I want to consider the first row, if same comes in the given range i want to... (1 Reply)
Discussion started by: gopikgunda
1 Replies

2. Shell Programming and Scripting

remove duplicates within a block in a file..help required

hi.. i have a file in the following format :- name-a age -12 address-123 age-12 phone-22222 ============ name-ab age -11 address-123 age-11 phone-222223 ============= name-abc age -12 address-1234 age-12 phone-2222223 ============= (2 Replies)
Discussion started by: nipun_garg
2 Replies

3. Shell Programming and Scripting

Remove duplicates from end of file

1/p ---- A B C A C o/p --- B A C From input file it should remove duplicates from end without changing order (5 Replies)
Discussion started by: lavnayas
5 Replies

4. Shell Programming and Scripting

Remove duplicates from a file

Hi, I need to remove duplicates from a file. The file will be like this 0003 10101 20100120 abcdefghi 0003 10101 20100121 abcdefghi 0003 10101 20100122 abcdefghi 0003 10102 20100120 abcdefghi 0003 10103 20100120 abcdefghi 0003 10103 20100121 abcdefghi Here if the first colum and... (6 Replies)
Discussion started by: gpaulose
6 Replies

5. Shell Programming and Scripting

Filter/remove duplicate .dat file with certain criteria

I am a beginner in Unix. Though have been asked to write a script to filter(remove duplicates) data from a .dat file. File is very huge containig billions of records. contents of file looks like 30002157,40342424,OTC,mart_rec,100, ,0 30002157,40343369,OTC,mart_rec,95, ,0... (6 Replies)
Discussion started by: mukeshguliao
6 Replies

6. Shell Programming and Scripting

Remove interspersed headers in .dat file with AWK

Heya there, A small selection of my data is shown below. DATE TIME FRAC_DAYS_SINCE_JAN1 2011-06-25 08:03:20.000 175.33564815 2011-06-25 08:03:25.000 175.33570602 2011-06-25 ... (4 Replies)
Discussion started by: gd9629
4 Replies

7. Shell Programming and Scripting

Remove <CR><LF> from the dat file in unix

Hi, The source system has created the file in the dat format and put into the linux directory as mentioned below. I want to do foloowing things. a) Delete the Line started with <CR><LF> in the record b)Also line ...........................................................<CR><LF> ... (1 Reply)
Discussion started by: mr_harish80
1 Replies

8. UNIX for Dummies Questions & Answers

Remove duplicates from a file

Can u tell me how to remove duplicate records from a file? (11 Replies)
Discussion started by: saga20
11 Replies

9. UNIX for Dummies Questions & Answers

Remove duplicates and keep them in a separate file

Hi, I have a tablular separated file and I want to remove all the rows that have duplicates. The diuplicates I need to check are in column 13. I have tried to use awk but I have no Idea how to keep the duplicate file. awk 'FNR==NR{a++;next}(a> 1)' tomodify.txt tomodify.txt > new.txt ... (4 Replies)
Discussion started by: flacchy
4 Replies

10. UNIX for Advanced & Expert Users

Remove duplicates in flat file

Hi all, I have a issues while loading a flat file to the DB. It is taking much time. When analyzed i found out that there are duplicates entry in the flat file. There are 2 type of Duplicate entry. 1) is entire row is duplicate. ( i can use sort | uniq) to remove the duplicated entry. 2) the... (4 Replies)
Discussion started by: samjoshuab
4 Replies
load_datafile(3alleg4)						  Allegro manual					    load_datafile(3alleg4)

NAME
load_datafile - Loads a datafile into memory. Allegro game programming library. SYNOPSIS
#include <allegro.h> DATAFILE *load_datafile(const char *filename); DESCRIPTION
Loads a datafile into memory in one go. If the datafile has been encrypted, you must first call packfile_password() to set the appropriate key. If the datafile contains truecolor graphics, you must set the video mode or call set_color_conversion() before loading it. Example: /* Load the resources for our game. */ DATAFILE *dat = load_datafile("game.dat"); if (!dat) abort_on_error("Couldn't load sound resources!"); /* Use resources. */ ... /* Destroy them when we don't need them any more. */ unload_datafile(dat); RETURN VALUE
Returns a pointer to the DATAFILE, or NULL on error. Remember to free this DATAFILE later to avoid memory leaks. SEE ALSO
load_datafile_callback(3alleg4), unload_datafile(3alleg4), load_datafile_object(3alleg4), set_color_conversion(3alleg4), fixup_datafile(3alleg4), packfile_password(3alleg4), find_datafile_object(3alleg4), register_datafile_object(3alleg4), excustom(3alleg4), exdata(3alleg4), exexedat(3alleg4), exgui(3alleg4), exsprite(3alleg4), exunicod(3alleg4) Allegro version 4.4.2 load_datafile(3alleg4)
All times are GMT -4. The time now is 02:54 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy