Removing dupes within 2 delimited areas in a large dictionary file Post: 302740273

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Issue with Removing Carriage Return (^M) in delimited file

Hi - I tried to remove ^M in a delimited file using "tr -d "\r" and "sed 's/^M//g'", but it does not work quite well. While the ^M is removed, the format of the record is still cut in half, like a,b, c c,d,e The delimited file is generated using sh script by outputing a SQL query result to...

2. Shell Programming and Scripting

Removing blanks in a text tab delimited file

Hi Experts I am very new to perl and need to make a script using perl. I would like to remove blanks in a text tab delimited file in in a specfic column range ( colum 21 to column 43) sample input and output shown below : Input: 117 102 650 652 654 656 117 93 95...

3. Shell Programming and Scripting

Removing Embedded Newline from Delimited File

Hey there - a bit of background on what I'm trying to accomplish, first off. I am trying to load the data from a pipe delimited file into a database. The loading tool that I use cannot handle embedded newline characters within a field, so I need to scrub them out. Solutions that I have tried...

4. Shell Programming and Scripting

Large pipe delimited file that I need to add CR/LF every n fields

I have a large flat file with variable length fields that are pipe delimited. The file has no new line or CR/LF characters to indicate a new record. I need to parse the file and after some number of fields, I need to insert a CR/LF to start the next record. Input file ...

5. Shell Programming and Scripting

Extracting a portion of data from a very large tab delimited text file

Hi All I wanted to know how to effectively delete some columns in a large tab delimited file. I have a file that contains 5 columns and almost 100,000 rows 3456 f g t t 3456 g h 456 f h 4567 f g h z 345 f g 567 h j k lThis is a very large data file and tab delimited. I need...

6. Shell Programming and Scripting

Script Optimization - large delimited file, for loop with many greps

Since there are approximately 75K gsfiles and hundreds of stfiles per gsfile, this script can take hours. How can I rewrite this script, so that it's much faster? I'm not as familiar with perl but I'm open to all suggestions. ls file.list>$split for gsfile in `cat $split`; do csplit...

7. Shell Programming and Scripting

Removing Dupes from huge file- awk/perl/uniq

Hi, I have the following command in place nawk -F, '!a++' file > file.uniq It has been working perfectly as per requirements, by removing duplicates by taking into consideration only first 3 fields. Recently it has started giving below error: bash-3.2$ nawk -F, '!a++'...

8. Shell Programming and Scripting

Merging dupes on different lines in a dictionary

I am working on a homonym dictionary of names i.e. names which are clustered together according to their �sound-alike� pronunciation: An example will make this clear: Since the dictionary is manually constructed it often happens that inadvertently two sets of �homonyms� which should be grouped...

9. UNIX for Advanced & Expert Users

Need optimized awk/perl/shell to give the statistics for the Large delimited file

I have a file size is around 24 G with 14 columns, delimiter with "|" My requirement- can anyone provide me the fastest and best to get the below results Number of records of the file First column and second Column- Unique counts Thanks for your time Karti ------ Post updated at...

10. Shell Programming and Scripting

Remove dupes in a large file

I have a large file 1.5 gb and want to sort the file. I used the following AWK script to do the job !x++ The script works but it is very slow and takes over an hour to do the job. I suspect this is because the file is not sorted. Any solution to speed up the AWk script or a Perl script would...

LEARN ABOUT SUSE

dpbindic

DPBINDIC(1)						      General Commands Manual						       DPBINDIC(1)

NAME

       dpbindic - Convert a binary-form dictionary into a text-form dictionary

SYNOPSYS

       dpbindic [ -xiu [ frequency ] ] binary-file [ text-file ]

DESCRIPTION

       dpbindic  outputs the file information of the binary-form dictionary file specified in binary-file .  At this time, the word information of
       the dictionary can be output in text form to the standard output.  To do so, use test-file to specify the text-form dictionary used as  the
       source  of  binary-form	dictionary  file.  If this specification is omitted, the text dictionary file information in the binary dictionary
       file will be output.  The standard grammar file name is /usr/local/canna/lib/dic/hyoujun.gram. It will be used if  the  grammar	file  name
       specification is omitted.  The output format of word information data is specified using an option.

OPTIONS

       -x     Outputs the data without using omission symbol @, which is used when the initial word represents the reading.

       -i     Replaces the reading and word for output.

       -u     Outputs  the  candidates	used in conversion.  Outputs all candidates having frequency or more.  If frequency is omitted, all candi-
	      dates having frequency 1 will be output.

EXAMPLES

       (1) If the text-form dictionary file name is omitted:
       %dibindic iroha.cbd
       (Text dictionary file name = Directory size + Word size, packed)
       iroha.swd = 2985 + 5306 pak a4
       iroha.mwd = 36276 + 113139 pak a4

       (2) If the text-form dictionary file name iroha.mwd is specified:
       %dpbindic iroha.cbd iroha.mwd
       (Text dictionary file name = Directory size + Word size, packed)
       iroha.mwd = 36276 + 113139 pak a4

SEE ALSO

       mkbindic(1), dicar(1)

																       DPBINDIC(1)