Dynamic Webserver + special characters (ASCII)

Kleriker · Post by **Kleriker** » Wed Jan 05, 2011 5:35 pm

Hey guys,

i have a big problem and i am in need of your help. The dynamic webserver is not able to handle special german characters (like äüö) and eventghost only delivers error messages "'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128))". I use dynamic variables which could contain these special characters, so i am not able to just change the html code. These text-elements (from the variables) should get posted on the website (i use a callmonitor).

I hope somebody could help me, because i have no idea how to solve this problem. I really would appreciate your help

Best regards,
Alex

stottle · Post by **stottle** » Wed Jan 05, 2011 8:41 pm

I think this post is relevant. I tried posting accented characters into the html file directly and got invalid data when my editor (notepad++) had the encoding set to ascii, but the characters displayed correctly when I changed the encoding the utf-8.

Then I tried using a parameter (I manually set eg.result = u"äüö") and put {{eg.result}} in the html. With the .html file set to ascii, I was getting an error, with the encoding set to utf-8 the accented characters displayed on the webpage.

[edit]This was with my fork, but I'm pretty sure the code is the same in this respect.[/edit]

Brett

Kleriker · Post by **Kleriker** » Wed Jan 05, 2011 9:32 pm

Hi,

thank you for your answer, but it didn´t help

I converted the html into utf-8 but eventghost still showes error messages. Any idea ?

Best regards,
Alex

stottle · Post by **stottle** » Wed Jan 05, 2011 9:43 pm

You need to isolate the problem. I'd suggest you
1) Paste the accented characters into the .html and make sure they print directly
2) Put something like {{eg.globals.testVar}} in the .html and open the python shell (help->python shell, IIRC). At the python shell type, >> eg.globals.testVar = u"äüö" (don't type the ">>"). Then open the page and see if that works.

If those both work, you need to let us know how you are setting the variable you are trying to get. I don't know what a "callmonitor" is, if that is what you are trying to set the variable with.

Brett

Kleriker · Post by **Kleriker** » Wed Jan 05, 2011 9:55 pm

Hi,

ok i just found out, that at the moment i converted the html file to utf-8 the error-messages show up even when my variable is just "123".
Maybe i have a bigger problem ?!

Best regards,
Alex

stottle · Post by **stottle** » Wed Jan 05, 2011 10:17 pm

Can you try the steps I posted?

Also, you aren't showing us any the the html you are using, nor how you are setting the variables you want to access, so I have no idea if you have a bigger problem or not.

Brett

Kleriker · Post by **Kleriker** » Wed Jan 05, 2011 10:26 pm

You are right ... i am sorry.

Here the html:

Code: Select all

<html>
    <head></head>
    <body>
        <p>
            Umlaute: {{eg.globals.test}}
        </p>
    </body>
</html>

and here the code for the variable:

Code: Select all

eg.globals.test = "13"

As i said, when i convert the html to utf-8 it is followed by these error messages:

Code: Select all

Webserver socket error /test2.html
<type 'exceptions.Exception'> ('ascii', u'\ufeff<html>\n    <head></head>\n    <body>\n        <p>\n            Umlaute: \n        </p>\n    </body>\n</html>', 0, 1, 'ordinal not in range(128)')
----------------------------------------
....etc....
the last error:
UnicodeEncodeError: 'ascii' codec can't encode character u'\ufeff' in position 0: ordinal not in range(128)

stottle · Post by **stottle** » Wed Jan 05, 2011 11:24 pm

What program did you use to save to utf-8? Also, are you sure you saved to utf-8?

The '\uffef' at the beginning of the file is (I think) the unicode BOM for utf-16, not utf-8. Either way, it is that, not the {{eg.globals.test}} that is causing the problem. So I think that problem will be fixed if you can get the encoding straightened out. Can you try saving the file again, or creating a new file?

I suggest you make sure you can load a page without dynamic parsing first or any accented characters first. Then I would add accented characters statically to the file 2nd. Once you know you can load static utf-8 characters, then I would try the dynamic parsing ("{{eg.globals.test}}")

Brett

miljbee · Post by **miljbee** » Thu Jan 06, 2011 9:43 am

Hello,

like Kleriker, I am not able to get special chars working with the advanced webserver.
As I saw this post, I thought it was time to test a little bit more.

It still don't work, here is what I have tested :
I have created 3 text files with Notepad++ :
ansi.html : format is set to ansi in np++
utf8.html : format is set to utf8 (without BOM) in np++
utf82.html : format is set to utf8 in np++

the content of these 3 files is identical and very simple, just 2 lines, no ref to python vars :

Code: Select all

test
tést

EG Log with ansi.html :

Code: Select all

2011-01-06 10:28:00  Webserver socket error /ansi.html
2011-01-06 10:28:00  <type 'exceptions.Exception'> ('utf8', 'test\r\nt\xe9st', 7, 10, 'invalid data')
2011-01-06 10:28:00  ----------------------------------------
2011-01-06 10:28:00  Exception happened during processing of request from ('127.0.0.1', 2676)
2011-01-06 10:28:00  Traceback (most recent call last):
2011-01-06 10:28:00  File "SocketServer.pyc", line 558, in process_request_thread
2011-01-06 10:28:00  File "SocketServer.pyc", line 320, in finish_request
2011-01-06 10:28:00  File "SocketServer.pyc", line 615, in __init__
2011-01-06 10:28:00  File "BaseHTTPServer.pyc", line 329, in handle
2011-01-06 10:28:00  File "BaseHTTPServer.pyc", line 323, in handle_one_request
2011-01-06 10:28:00  File "C:\Program Files\EventGhost\plugins\Webserver\__init__.py", line 194, in do_GET
2011-01-06 10:28:00  File "C:\Program Files\EventGhost\plugins\Webserver\__init__.py", line 149, in SendContent
2011-01-06 10:28:00  File "jinja2\environment.pyc", line 552, in get_template
2011-01-06 10:28:00  File "jinja2\loaders.pyc", line 96, in load
2011-01-06 10:28:00  File "C:\Program Files\EventGhost\plugins\Webserver\__init__.py", line 69, in get_source
2011-01-06 10:28:00  File "encodings\utf_8.pyc", line 16, in decode
2011-01-06 10:28:00  UnicodeDecodeError: 'utf8' codec can't decode bytes in position 7-9: invalid data
2011-01-06 10:28:00  ----------------------------------------

EG Log with utf8.html :

Code: Select all

2011-01-06 10:30:25  Webserver socket error /utf8.html
2011-01-06 10:30:25  <type 'exceptions.Exception'> ('ascii', u'test\nt\xe9st', 6, 7, 'ordinal not in range(128)')
2011-01-06 10:30:25  ----------------------------------------
2011-01-06 10:30:25  Exception happened during processing of request from ('127.0.0.1', 2701)
2011-01-06 10:30:25  Traceback (most recent call last):
2011-01-06 10:30:25  File "SocketServer.pyc", line 558, in process_request_thread
2011-01-06 10:30:25  File "SocketServer.pyc", line 320, in finish_request
2011-01-06 10:30:25  File "SocketServer.pyc", line 615, in __init__
2011-01-06 10:30:25  File "BaseHTTPServer.pyc", line 329, in handle
2011-01-06 10:30:25  File "BaseHTTPServer.pyc", line 323, in handle_one_request
2011-01-06 10:30:25  File "C:\Program Files\EventGhost\plugins\Webserver\__init__.py", line 194, in do_GET
2011-01-06 10:30:25  File "C:\Program Files\EventGhost\plugins\Webserver\__init__.py", line 160, in SendContent
2011-01-06 10:30:25  File "socket.pyc", line 292, in write
2011-01-06 10:30:25  UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 6: ordinal not in range(128)
2011-01-06 10:30:25  ----------------------------------------

EG Log with utf82.html :

Code: Select all

2011-01-06 10:32:33  Webserver socket error /utf82.html
2011-01-06 10:32:33  <type 'exceptions.Exception'> ('ascii', u'\ufefftest\nt\xe9st', 0, 1, 'ordinal not in range(128)')
2011-01-06 10:32:33  ----------------------------------------
2011-01-06 10:32:33  Exception happened during processing of request from ('127.0.0.1', 2706)
2011-01-06 10:32:33  Traceback (most recent call last):
2011-01-06 10:32:33  File "SocketServer.pyc", line 558, in process_request_thread
2011-01-06 10:32:33  File "SocketServer.pyc", line 320, in finish_request
2011-01-06 10:32:33  File "SocketServer.pyc", line 615, in __init__
2011-01-06 10:32:33  File "BaseHTTPServer.pyc", line 329, in handle
2011-01-06 10:32:33  File "BaseHTTPServer.pyc", line 323, in handle_one_request
2011-01-06 10:32:33  File "C:\Program Files\EventGhost\plugins\Webserver\__init__.py", line 194, in do_GET
2011-01-06 10:32:33  File "C:\Program Files\EventGhost\plugins\Webserver\__init__.py", line 160, in SendContent
2011-01-06 10:32:33  File "socket.pyc", line 292, in write
2011-01-06 10:32:33  UnicodeEncodeError: 'ascii' codec can't encode character u'\ufeff' in position 0: ordinal not in range(128)
2011-01-06 10:32:33  ----------------------------------------

I don't know what this really means, I will try to search more, let me know if you have ideas !

Regards

miljbee · Post by **miljbee** » Thu Jan 06, 2011 10:47 am

in a python shell :

Code: Select all

>>>print u'test\nt\xe9st'
test
tést
>>>print str(u'test\nt\xe9st')
Traceback (most recent call last):
  File "<input>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 6: ordinal not in range(128)

Don't know if it helps ...

Kleriker · Post by **Kleriker** » Thu Jan 06, 2011 11:33 am

Hey guys,

ok i found out that i had to convert the html file to "UTF-8 without BOM". Btw. i use Notepad++.

But when my html code looks like this:

<html>
<head></head>
<body>
Umlaute: 12ä
</body>
</html>

it stops working again, followed by error messages in EG.

Also i found out that the .py file from the dynamic-webserver plugin is coded as ANSI. Is this normal ?

Best regards,
Alex

stottle · Post by **stottle** » Sun Jan 09, 2011 5:03 pm

Sorry I haven't posted anything back. I'm moving in two days, so I haven't had any time for this. I'll need to take a look at what is different between my fork and the .py file for dynamic webserver, since I can load a simple .html file with unicode successfully in my fork. But unfortunately it is going to have to wait until I get settled a bit.

miljbee - your python shell tests make sense. u'???' is unicode, str() tries to convert the unicode to ascii. Python 2 treats unicode and ascii as two different types. The conversion will throw if you have unicode you don't tell python how to convert.

Brett

Kleriker · Post by **Kleriker** » Sun Jan 09, 2011 5:49 pm

Hi Brett

No problem at all. I am not in a hurry. Quick question: what do you mean by "fork" ?

Best regards and good luck with your move,
Alex

stottle · Post by **stottle** » Sun Jan 09, 2011 6:15 pm

Kleriker wrote:Quick question: what do you mean by "fork" ?

I'm developing a separate project based on EG called SatoriHub. It's still at alpha-level development (pre-beta, that is) and the website needs a bit of work, but this is the website.

My opinion is that maybe 1 in 50 people that would be interested in EG have the knowledge and/or patience to create a custom configuration and deal with things like exclusion macros. So I'm putting additional processing into the "core" to simplify setup and configuration. It seemed like a good idea before Bitmonster quit EG development, and seems like an even better idea since....

Brett

Kleriker · Post by **Kleriker** » Sun Jan 09, 2011 6:23 pm

Woow that´s just great. I was a little bit afraid, that the EG-projekt will die with bitmonster gone, but there is a new hope with your program. Thanks for that. I will have a closer look the next days.

Best regards,
Alex

EventGhost

Dynamic Webserver + special characters (ASCII)

Dynamic Webserver + special characters (ASCII)

Re: Dynamic Webserver + special characters (ASCII)

Re: Dynamic Webserver + special characters (ASCII)

Re: Dynamic Webserver + special characters (ASCII)

Re: Dynamic Webserver + special characters (ASCII)

Re: Dynamic Webserver + special characters (ASCII)

Re: Dynamic Webserver + special characters (ASCII)

Re: Dynamic Webserver + special characters (ASCII)

Re: Dynamic Webserver + special characters (ASCII)

Re: Dynamic Webserver + special characters (ASCII)

Re: Dynamic Webserver + special characters (ASCII)

Re: Dynamic Webserver + special characters (ASCII)

Re: Dynamic Webserver + special characters (ASCII)

Re: Dynamic Webserver + special characters (ASCII)

Re: Dynamic Webserver + special characters (ASCII)