Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Perl script to sort an Excel file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Perl script to sort an Excel file
# 1  
Old 01-13-2010
Question Perl script to sort an Excel file
Hello!

I need to sort a file that is partly in English partly in Bulgarian.

The original file is an Excel file but I converted it to a tab-delimited text file. The encoding of the tab delimited file is UTF-8.

To sort the text, the script should test every line of the text file to see if there are any Cyrillic UTF-8 characters.

Even if a single Cyrillic character is found, it will be quite enough to conclude that the line is in Bulgarian. If no Cyrillic character is found, then the text should be in English.

The UTF-8 range of Cyrillic characters is U+0400 to U+04FF

After finding a Cyrillic character the script should add at the end the same line Tab+'bg'; when no Cyrillic character is found in the line, Tab+'en' should be added to the end of the line. I need the tabs so that I can open the file back in Excel and sort the lines.

I believe the script should be best written in Perl.

My system is MS Windows XP Professional. I have cygwin+Perl and Strawberry Perl installed.

Thanks in advance!
Last edited by degoor; 01-13-2010 at 07:49 AM.. Reason: Missed word
# 2  
Old 01-13-2010
Code:
awk '{ print $0 "\t" (/[\320-\323]/ ? "bg": "en")}' data

The range U+0400 to U+04FF when encoded in UTF-8 will always be a two byte sequence, the first of which must have a value between 0320-0323 (octal), 0xD0-0xD3 (hex), 208-211 (decimal). Since this byte range serves no other purpose, its presence indicates a character in the relevant range.

I realize you requested a perl solution, but I'm not much of a perl hacker. If you have cygwin, you should have awk (gawk, most likely).

Regards,
alister
Last edited by alister; 01-13-2010 at 11:52 AM.. Reason: Elaboration
# 3  
Old 01-13-2010
Question
alister, thank you very much for you help!

It seems though that the range of the Cyrillic characters is different because I am getting all lines marked with "en" which is not true.

I tried your code having coded the text in my editor 'UTF-8', 'UTF-8+BOM' and 'Unicode' but neither works - all lines got marked with "en".

Actually as far as I have this job done I don't mind any mean.

I requested Perl because it is considered to have an excellent support for Unicode and regular expressions.

Can I try your code for some other range if I change the encoding to ANSI?
# 4  
Old 01-13-2010
Hello, degoor:

I tested the code on a text file containing bulgarian (utf-8 encoded) and english, and the code worked correctly. I inspected the byte values using hexdump (od would do) and they confirmed the presence of two byte sequences beginning with 0320-0323 octal. I also double checked your byte ranges using wikipedia/google and U+0400 through U+04FF are indeed cyrillic.

I suggest taking a look at the file using od or hexdump (or a windows hexeditor) to see exactly what's in the file you intend to process with perl/awk.

My sample utf-8 encoded text with cyrillic came from the Columbia UTF-8 sampler, top result when googling "utf-8 sampler". You can test my code against that to confirm that there isn't some other malfunction taking place (perhaps copy-paste mangling, etc).

alister
# 5  
Old 01-13-2010
Quote:
Originally Posted by alister
Code:
awk '{ print $0 "\t" (/[\320-\323]/ ? "bg": "en")}' data

The range U+0400 to U+04FF when encoded in UTF-8 will always be a two byte sequence, the first of which must have a value between 0320-0323 (octal), 0xD0-0xD3 (hex), 208-211 (decimal). Since this byte range serves no other purpose, its presence indicates a character in the relevant range.

I realize you requested a perl solution, but I'm not much of a perl hacker. If you have cygwin, you should have awk (gawk, most likely).

Regards,
alister
Actually I run your code in cygwin bash, like this:

Code:
gawk '{ print $0 "\t" (/[\320-\323]/ ? "bg": "en")}' source_txt_ file > result_txt_file

Is that what you had in mind?
Last edited by degoor; 01-13-2010 at 04:25 PM.. Reason: Spelling
# 6  
Old 01-13-2010
That looks fine, except for what I assume is a typo: the space between "source_txt_" and "file"

To be certain, I tested using nawk/osx and gawk/cygwin-winxp; both worked as expected.

It would help if you would share a reduced version of your source file, one english line and one bulgarian line (obfuscating any sensitive info, of course). Also, given the strange things that can happen in transit, a paste of a hex/octal dump of those bytes would be very helpful.

Example:

Code:
$ cat utf8-data 
English: I can eat glass
Bulgarian: Мога да ям стъкло, то не

$ od -c utf8-data 
0000000    E   n   g   l   i   s   h   :       I       c   a   n       e
0000020    a   t       g   l   a   s   s  \n   B   u   l   g   a   r   i
0000040    a   n   :     320 234 320 276 320 263 320 260     320 264 320
0000060  260     321 217 320 274     321 201 321 202 321 212 320 272 320
0000100  273 320 276   ,     321 202 320 276     320 275 320 265  \n    
0000117

$ awk '{ print $0 "\t" (/[\320-\323]/ ? "bg": "en")}' utf8-data
English: I can eat glass        en
Bulgarian: Мога да ям стъкло, то не     bg

Regards,
alister
# 7  
Old 01-14-2010
alister:

Here is a sample file:
Code:
TextId	TextDescription	Explanation	Signature	Translator Question	Customer Answer	Fake
3	Комуникационен дефект!					1
10	Проверката е приключила, системата е ОК					1
14	Искате ли да изтриете? 					1
4655	Add key		MaJ			1
4656	Module		MaJ			1
4657	Available		MaJ			1

The file is encoded UTF-8 by my editor EditPlus.

There are 7 lines in the file:
The 1st line consists of the column titles (so it is actually in English),
The next 3 line are in Bulgarian,
The last 3 lines are in English.

I got all lines marked as "en" Smilie

The problem might be in the settings of cygwin (I use defaults) so I shall try to set myself up some Linux system (Ubuntu probably Smilie) and test the code on it.

@Franklin52: Sorry I didn't know that I can't post direct links here. I do apologise for that.

@alister: Please use the file from the link that I sent you as a PM. It should be same as the above but still ... Smilie
Last edited by degoor; 01-14-2010 at 04:42 AM.. Reason: Link removed, adding attached file from link
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to sort a column in excel/csv file?

I have to sort the 4th column of an excel/csv file. I tried the following command sort -u --field-separator=, --numeric-sort -k 2 -n dinesh.csv > test.csv But, it's not working. Moreover, I have to do the same for more than 30 excel/csv file. So please help me to do the same. (6 Replies)
Discussion started by: dineshkumarsrk
6 Replies

2. Shell Programming and Scripting

Perl script to fill the entire row of Excel file with color based on pattern match

Hi All , I have to write one Perl script in which I need to read one pre-existing xls and based on pattern match for one word in some cells of the XLS , I need to fill the entire row with one color of that matched cell and write the content to another excel Please find the below stated... (2 Replies)
Discussion started by: kshitij
2 Replies

3. Shell Programming and Scripting

Tabbed multiple csv files into one single excel file with using shell script not perl

Hi Experts, I am querying backup status results for multiple databases and getting each and every database result in one csv file. so i need to combine all csv files in one excel file with separate tabs. I am not familiar with perl script so i am using shell script. Could anyone please... (4 Replies)
Discussion started by: ramakrk2
4 Replies

4. Shell Programming and Scripting

Script to generate Excel file or to SQL output data to Excel format/tabular format

Hi , i am generating some data by firing sql query with connecting to the database by my solaris box. The below one should be the header line of my excel ,here its coming in separate row. TO_CHAR(C. CURR_EMP_NO ---------- --------------- LST_NM... (6 Replies)
Discussion started by: dani1234
6 Replies

5. Shell Programming and Scripting

Perl script to Merge contents of 2 different excel files in a single excel file

All, I have an excel sheet Excel1.xls that has some entries. I have one more excel sheet Excel2.xls that has entries only in those cells which are blank in Excel1.xls These may be in different workbooks. They are totally independent made by 2 different users. I have placed them in a... (1 Reply)
Discussion started by: Anamika08
1 Replies

6. Shell Programming and Scripting

Writing excel file using perl : Excel file formatting changed

I am trying to create a program where user can input data in certain excel cells using user interface on internet....the programming is on perl and server is unix But when i parse data into excel the formatting of sheets is turned to default and all macro coding removed. What to do...Please... (7 Replies)
Discussion started by: mud_born
7 Replies

7. UNIX and Linux Applications

Perl Script to read an excel file into an array and search in the UNIX directories

Hi, I want the Perl script with versions 5.8.2 and 5.8.5 starting with #!/usr/bin/perl The Perl program should read the excel file or text file line by line and taking into an array and search in the UNIX directories for reference file of .jsp or .js or .xsl with path .The Object names... (2 Replies)
Discussion started by: pasam
2 Replies

8. Shell Programming and Scripting

Sort Excel File

Hi, I am new to scripting. I have a requirement 1.convert excel 2007 format to excel 2003 fromat 2.sort the excel file -( all columns in descending order). This needs to be done progarmmatically. Please let me know what would be the possible steps that I would need to take inorder to... (2 Replies)
Discussion started by: eva13
2 Replies

9. Shell Programming and Scripting

How to sort columns in excel(csv) file

i want sort columns with headers based on another file headers file1 eg: i'm having an empty file with only coumn names like lastname firstname title expirydate stlcno status etc... another file with same column names and some other as well but in different order... file2 eg:firstname... (2 Replies)
Discussion started by: Man83Nagesh
2 Replies

10. Shell Programming and Scripting

PERL: Split Excel Workbook to Indiv Excel files

Hi, I am trying to find a way to read an excel work book with multiple worksheets. And write each worksheet into a new excel file using perl. My environment is Unix. For example: I have an excel workbook TEST.xls and it has Sheet1, Sheet2, Sheet3 worksheets. I would like to create... (2 Replies)
Discussion started by: sandeep78
2 Replies
Login or Register to Ask a Question