I need to sort a file that is partly in English partly in Bulgarian.
The original file is an Excel file but I converted it to a tab-delimited text file. The encoding of the tab delimited file is UTF-8.
To sort the text, the script should test every line of the text file to see if there are any Cyrillic UTF-8 characters.
Even if a single Cyrillic character is found, it will be quite enough to conclude that the line is in Bulgarian. If no Cyrillic character is found, then the text should be in English.
The UTF-8 range of Cyrillic characters is U+0400 to U+04FF
After finding a Cyrillic character the script should add at the end the same line Tab+'bg'; when no Cyrillic character is found in the line, Tab+'en' should be added to the end of the line. I need the tabs so that I can open the file back in Excel and sort the lines.
I believe the script should be best written in Perl.
My system is MS Windows XP Professional. I have cygwin+Perl and Strawberry Perl installed.
Thanks in advance!
Last edited by degoor; 01-13-2010 at 07:49 AM..
Reason: Missed word
The range U+0400 to U+04FF when encoded in UTF-8 will always be a two byte sequence, the first of which must have a value between 0320-0323 (octal), 0xD0-0xD3 (hex), 208-211 (decimal). Since this byte range serves no other purpose, its presence indicates a character in the relevant range.
I realize you requested a perl solution, but I'm not much of a perl hacker. If you have cygwin, you should have awk (gawk, most likely).
Regards,
alister
Last edited by alister; 01-13-2010 at 11:52 AM..
Reason: Elaboration
I tested the code on a text file containing bulgarian (utf-8 encoded) and english, and the code worked correctly. I inspected the byte values using hexdump (od would do) and they confirmed the presence of two byte sequences beginning with 0320-0323 octal. I also double checked your byte ranges using wikipedia/google and U+0400 through U+04FF are indeed cyrillic.
I suggest taking a look at the file using od or hexdump (or a windows hexeditor) to see exactly what's in the file you intend to process with perl/awk.
My sample utf-8 encoded text with cyrillic came from the Columbia UTF-8 sampler, top result when googling "utf-8 sampler". You can test my code against that to confirm that there isn't some other malfunction taking place (perhaps copy-paste mangling, etc).
The range U+0400 to U+04FF when encoded in UTF-8 will always be a two byte sequence, the first of which must have a value between 0320-0323 (octal), 0xD0-0xD3 (hex), 208-211 (decimal). Since this byte range serves no other purpose, its presence indicates a character in the relevant range.
I realize you requested a perl solution, but I'm not much of a perl hacker. If you have cygwin, you should have awk (gawk, most likely).
Regards,
alister
Actually I run your code in cygwin bash, like this:
Is that what you had in mind?
Last edited by degoor; 01-13-2010 at 04:25 PM..
Reason: Spelling
That looks fine, except for what I assume is a typo: the space between "source_txt_" and "file"
To be certain, I tested using nawk/osx and gawk/cygwin-winxp; both worked as expected.
It would help if you would share a reduced version of your source file, one english line and one bulgarian line (obfuscating any sensitive info, of course). Also, given the strange things that can happen in transit, a paste of a hex/octal dump of those bytes would be very helpful.
Here is a sample file:
The file is encoded UTF-8 by my editor EditPlus.
There are 7 lines in the file:
The 1st line consists of the column titles (so it is actually in English),
The next 3 line are in Bulgarian,
The last 3 lines are in English.
I got all lines marked as "en"
The problem might be in the settings of cygwin (I use defaults) so I shall try to set myself up some Linux system (Ubuntu probably ) and test the code on it.
@Franklin52: Sorry I didn't know that I can't post direct links here. I do apologise for that.
@alister: Please use the file from the link that I sent you as a PM. It should be same as the above but still ...
Last edited by degoor; 01-14-2010 at 04:42 AM..
Reason: Link removed, adding attached file from link
I have to sort the 4th column of an excel/csv file. I tried the following command
sort -u --field-separator=, --numeric-sort -k 2 -n dinesh.csv > test.csv
But, it's not working. Moreover, I have to do the same for more than 30 excel/csv file. So please help me to do the same. (6 Replies)
Hi All ,
I have to write one Perl script in which I need to read one pre-existing xls and based on pattern match for one word in some cells of the XLS , I need to fill the entire row with one color of that matched cell and write the content to another excel
Please find the below stated... (2 Replies)
Hi Experts,
I am querying backup status results for multiple databases and getting each and every database result in one csv file. so i need to combine all csv files in one excel file with separate tabs. I am not familiar with perl script so i am using shell script.
Could anyone please... (4 Replies)
Hi ,
i am generating some data by firing sql query with connecting to the database by my solaris box.
The below one should be the header line of my excel ,here its coming in separate row.
TO_CHAR(C. CURR_EMP_NO
---------- ---------------
LST_NM... (6 Replies)
All,
I have an excel sheet Excel1.xls that has some entries.
I have one more excel sheet Excel2.xls that has entries only in those cells which are blank in Excel1.xls
These may be in different workbooks. They are totally independent made by 2 different users.
I have placed them in a... (1 Reply)
I am trying to create a program where user can input data in certain excel cells using user interface on internet....the programming is on perl and server is unix
But when i parse data into excel the formatting of sheets is turned to default and all macro coding removed.
What to do...Please... (7 Replies)
Hi,
I want the Perl script with versions 5.8.2 and 5.8.5 starting with
#!/usr/bin/perl
The Perl program should read the excel file or text file line by line and taking into an array and search in the UNIX directories for reference file of .jsp or .js or .xsl with path .The Object names... (2 Replies)
Hi,
I am new to scripting. I have a requirement
1.convert excel 2007 format to excel 2003 fromat
2.sort the excel file -( all columns in descending order).
This needs to be done progarmmatically. Please let me know what would be the possible steps that I would need to take inorder to... (2 Replies)
i want sort columns with headers based on another file headers
file1
eg: i'm having an empty file with only coumn names like
lastname firstname title expirydate stlcno status etc...
another file with same column names and some other as well but in different order...
file2
eg:firstname... (2 Replies)
Hi,
I am trying to find a way to read an excel work book with multiple worksheets.
And write each worksheet into a new excel file using perl. My environment is Unix.
For example: I have an excel workbook TEST.xls and it has Sheet1, Sheet2, Sheet3 worksheets. I would like to create... (2 Replies)