timeout pythonscript after some time?

If you have a question or need help, this is the place to be.
zen_SuR
Posts: 11
Joined: Mon Sep 17, 2018 5:12 pm

timeout pythonscript after some time?

Post by zen_SuR » Mon Sep 17, 2018 5:38 pm

Hello Community,

my problem seems simple:

i have a pythonscript function to read a php file via urlib2 periodically every 3 seconds (i set this up with the timer-plugin). sometimes it takes longer for various reasons to read. it still finishes after sometime and starts again - if it takes some time longer again for some reason it starts clocking up to the point where my internetconnection is theoretically working but i have to factory reset the modem to be able to surf in the internet... something with the dns-lookup seems going wrong at that point for all internetsites routet through the modem. i am not kidding... also from another device. I also cannot normally stop eventghost at that point and have to kill it via taskmanager. :)

easiest way would be to tell the function to stop if it takes longer than 2.5 seconds so the new cycle can start freshly.... so i kind of need to kill the thread and not just exit()?

i could just tell my script to use the timer to set up a new timer that triggers an action that ends the current one? how would that work? is it even possible to end a thread from outside? any other inputs? i put the code below:

"is_connected" is the function that looks if there is internet present.

the line:
todomi = urllib2.urlopen("https://data.josefwagner.net/intvid_tra ... iggers.php").read()

is the one that causes the mess...

Code: Select all


import urllib2
import string

def triggerlookup():
    print "triggerlookup started"
    eg.plugins.Timer.TimerAction(u'checktrigger', 0, 0, 3.0, u'check for triggers', False, True, 1, u'00:00:00')
    try: 
        print "try"
        todomi = urllib2.urlopen("https://data.josefwagner.net/intvid_transcreen/php-functions/active_triggers.php").read()
        print "trytrue"
    except urllib2.HTTPError, e:
        print "http error"
        raise SystemExit()
    except urllib2.URLError, e:
        print "url error"
        raise SystemExit()
    except Exception:
        print "exception_error_found in triggerlookup"
        raise SystemExit()
        
    if todomi:
            eg.TriggerEvent(u"command_from_dominique__"+str(todomi))
            print "triggerfiring"
            raise SystemExit()
            return True
            
    else:
            print "no trigger found"
            raise SystemExit()
            return False
        
import socket

def is_connected():
    try:
        # connect to the host -- tells us if the host is actually
        # reachable
        socket.create_connection(("www.google.com", 80))
        print "internet on"
        triggerlookup()
    except OSError:
        print "internet off"
        raise SystemExit()
    except Exception:
        print "internetcheck exception_error_found resetting timer to slower"
        eg.plugins.Timer.TimerAction(u'checktrigger', 0, 0, 10.0, u'check for triggers', False, True, 1, u'00:00:00')
        raise SystemExit()

is_connected()

raise SystemExit


zen_SuR
Posts: 11
Joined: Mon Sep 17, 2018 5:12 pm

Re: timeout pythonscript after some time?

Post by zen_SuR » Tue Sep 18, 2018 9:48 am

Hello,

eventhough the original question is still interesting ("how to really kill a running pythonscript from an outside macro so it does not continue in the background as a zombie-thread") i found the most elegant workaround which is also pretty simple for my concrete problem: basically i just have to turn off the timer as soon as the function starts and restart it when its done. in my case i also turn the timer to a slower intervall when an error occurs. it actually works and my internet runs smoothly even with slower intervals like 0.5 seconds... in case it still gets overwhelming (i tried intervals down to 0.1) it catches the problem soon enough before my modem crashes by setting the timer back to 10 seconds. if everything works than it goes back to its original interval iE 0.1.

I just did not think of it before because i was not how amazingly easy it was to use eventghostplugins in pythonscript-action.

here the modified script:

Code: Select all

import urllib2
import string




def triggerlookup():
    print "triggerlookup started"
    
    try:
        
        print "try"
        todomi = urllib2.urlopen("https://data.josefwagner.net/intvid_transcreen/php-functions/active_triggers.php").read()
        print "trytrue"
      
    except urllib2.HTTPError, e:
        print "http error setting timer to 10 seconds"
        eg.plugins.Timer.TimerAction(u'checktrigger', 0, 0, 10.0, u'check for triggers', False, True, 1, u'00:00:00')
        raise SystemExit
    except urllib2.URLError, e:
        print "url error setting timer to 10 seconds"
        eg.plugins.Timer.TimerAction(u'checktrigger', 0, 0, 10.0, u'check for triggers', False, True, 1, u'00:00:00')
        raise SystemExit
    except Exception:
        print "exception_error_found in triggerlookup - setting timer to 10 seconds"
        eg.plugins.Timer.TimerAction(u'checktrigger', 0, 0, 10.0, u'check for triggers', False, True, 1, u'00:00:00')
        raise SystemExit

       
    if todomi:
          

            eg.TriggerEvent(u"command_from_dominique__"+str(todomi))
            print "triggerfiring"
           
            
            
    else:
            print "no trigger found"
           
            
        
    eg.plugins.Timer.TimerAction(u'checktrigger', 0, 0, 3.0, u'check for triggers', False, True, 1, u'00:00:00')
    print "timer restarted"


import socket


def is_connected():
    
        try:
            # connect to the host -- tells us if the host is actually
            # reachable
            socket.create_connection(("www.google.com", 80))
            print "internet on"
            triggerlookup()
        except OSError:
            print "internet off"
            raise SystemExit()
        except Exception:
            print "internetcheck exception_error_found resetting timer to slower"
            eg.plugins.Timer.TimerAction(u'checktrigger', 0, 0, 10.0, u'check for triggers', False, True, 1, u'00:00:00')
            raise SystemExit()
    

    

eg.plugins.Timer.TimerAction(u'checktrigger', 4, 0, 0.0, u'check for triggers', False, True, 1, u'00:00:00')
print "timer stopped - starting function"
is_connected()

raise SystemExit

zen_SuR
Posts: 11
Joined: Mon Sep 17, 2018 5:12 pm

Re: timeout pythonscript after some time?

Post by zen_SuR » Wed Sep 19, 2018 10:20 am

So final update

my timerlogic made it better but did not fix the underlying problem. it just took longer to mess up and so i had to search again for the error and I FOUND IT.

the problem is urllib2 itself. it is simply is a mess and very very bad with proxies and dns-stuff. but there is another way. Instead of urllib2 i will always use "requests" now... my script now looks like this and is running without problems for a day now - so i am very confident that that was it:

Urllib2 is a mess and should not be used in any way unless you like weird errors. i hope this helps somebody somewhen so my updated working script:

Code: Select all

import requests
import string
import socket

def triggerlookup():
    print "triggerlookup started"
    print "try"
    todomi = requests.post('https://data.josefwagner.net/intvid_transcreen/php-functions/active_triggers.php', verify=False)
    print "trytrue"
    todomi = todomi.text
    print todomi
  
    if todomi:
        eg.TriggerEvent(u"command_from_dominique__"+str(todomi))
        print "triggerfiring"            
    else:
        print "no trigger found"

    eg.plugins.Timer.TimerAction(u'checktrigger', 0, 0, 3.0, u'check for triggers', False, True, 1, u'00:00:00')
    print "timer restarted"


import socket

def is_connected():
        try:
            # connect to the host -- tells us if the host is actually
            # reachable
            socket.create_connection(("www.google.com", 80))
            print "internet on"
            triggerlookup()
        except OSError:
            eg.plugins.Speech.TextToSpeech(u'IVONA 2 Emma OEM - British English female voice [22kHz]', -1, u'Alarm alarm alarm', u'', 100, u'Lautsprecher (Realtek High Definition Audio)')
            print "internet off"
            eg.plugins.Timer.TimerAction(u'checktrigger', 0, 0, 10.0, u'check for triggers', False, True, 1, u'00:00:00')
            raise SystemExit()
        except Exception:
            eg.plugins.Speech.TextToSpeech(u'IVONA 2 Emma OEM - British English female voice [22kHz]', -1, u'Alarm alarm alarm', u'', 100, u'Lautsprecher (Realtek High Definition Audio)')
            raise SystemExit()
        
eg.plugins.Timer.TimerAction(u'checktrigger', 4, 0, 0.0, u'check for triggers', False, True, 1, u'00:00:00')
print "timer stopped - starting function"
is_connected()
raise SystemExit


User avatar
kgschlosser
Site Admin
Posts: 5508
Joined: Fri Jun 05, 2015 5:43 am
Location: Rocky Mountains, Colorado USA

Re: timeout pythonscript after some time?

Post by kgschlosser » Wed Sep 19, 2018 10:36 pm

give this bad larry a try and see if it works for you

MAKE SURE TO READ THE DIRECTIONS

Code: Select all

# if you want this to be a self contained looping script then you need to create
# a macro add this script to a python script action. Then create an event or 
# drag and drop the Main.OnInit event into the macro. be sure the event is 
# before the action. This will automatically set the script up to only run when
# when EG starts up then loop until EG gets closed.

# This is the setting for how long to wait between loops. or the "interval"
# The interval is in seconds. so 3.5 is three and a half seconds

LOOP_INTERVAL = 5.0

# this is the timeout to wait for the server to respond. If you are running 
# this in a loop you can set this for however long you like. if this is going 
# to run by an event other then Main.OnInit triggering it ten you need to set 
# this to no greater then 3.0 seconds. this is the threshold where EG will 
# think that something got stuck and will want to close on ya. It is the same 
# time format as the LOOP_INTERVAL

# No website should ever take longer then a second to respond.

TIMEOUT = 1.0

URL = (
    "https://data.josefwagner.net/intvid_transcreen"
    "/php-functions/active_triggers.php"
)

import urllib2
import socket
import traceback

RUN_IN_LOOP = eg.event.string == 'Main.OnInit'


def exit():
    eg.PrintError(traceback.format_exc())

    if RUN_IN_LOOP:
        eg.scheduler.AddShortTask(LOOP_INTERVAL, get_request)
    else:
        eg.StopMacro()
        eg.Exit()


def get_request():
    try:
        request = urllib2.urlopen(URL, timeout=TIMEOUT)
    except urllib2.HTTPError:
        eg.PrintError('-- HTTP ERROR ' + ('-' * 25))
        exit()
    except urllib2.URLError:
        eg.PrintError('-- URL ERROR ' + ('-' * 26))
        exit()
    except socket.error:
        eg.PrintError('-- REQUEST TIMEOUT ' + ('-' * 20))
        exit()
    else:
        todomi = request.read()
        if todomi:
            eg.TriggerEvent(prefix="CommandFromDominique", suffix=todomi)
            
    if RUN_IN_LOOP:
        eg.scheduler.AddShortTask(LOOP_INTERVAL, get_request)

get_request()
If you like the work I have been doing then feel free to Image

User avatar
kgschlosser
Site Admin
Posts: 5508
Joined: Fri Jun 05, 2015 5:43 am
Location: Rocky Mountains, Colorado USA

Re: timeout pythonscript after some time?

Post by kgschlosser » Wed Sep 19, 2018 10:38 pm

you can also swap out urllib2 for requests. instead of using requests.post. you should use requests.get
If you like the work I have been doing then feel free to Image

zen_SuR
Posts: 11
Joined: Mon Sep 17, 2018 5:12 pm

Re: timeout pythonscript after some time?

Post by zen_SuR » Sun Sep 30, 2018 9:52 am

wow... it took me some time and i have found out that my last solution was just working better but did not resolve the underlying issue... i am trying out yours now and will report back if that fixed it - so far its looking good also with urllib2.... maybe i was too harsh with it :). we will see.

User avatar
kgschlosser
Site Admin
Posts: 5508
Joined: Fri Jun 05, 2015 5:43 am
Location: Rocky Mountains, Colorado USA

Re: timeout pythonscript after some time?

Post by kgschlosser » Sun Sep 30, 2018 10:18 am

requests uses urllib behind the scenes. it helps to remove some of the complexities of it. I personally like requests. but there are situations where urllib has to be used.
If you like the work I have been doing then feel free to Image

zen_SuR
Posts: 11
Joined: Mon Sep 17, 2018 5:12 pm

Re: timeout pythonscript after some time?

Post by zen_SuR » Sun Sep 30, 2018 9:38 pm

while yours is way more elegant it does not solve the underlying issue... my internet/modem breaks down after some time (here after about an hour) - no matter if i use requests or anything else... i am absolutly out of ideas. when it starts and i try to surf it works as long as i stay on already opened domains. if i want to resolve a new one it pretty much dies. any ideas?

this is my current code:

Code: Select all


# if you want this to be a self contained looping script then you need to create
# a macro add this script to a python script action. Then create an event or 
# drag and drop the Main.OnInit event into the macro. be sure the event is 
# before the action. This will automatically set the script up to only run when
# when EG starts up then loop until EG gets closed.

# This is the setting for how long to wait between loops. or the "interval"
# The interval is in seconds. so 3.5 is three and a half seconds

LOOP_INTERVAL = 3

# this is the timeout to wait for the server to respond. If you are running 
# this in a loop you can set this for however long you like. if this is going 
# to run by an event other then Main.OnInit triggering it ten you need to set 
# this to no greater then 3.0 seconds. this is the threshold where EG will 
# think that something got stuck and will want to close on ya. It is the same 
# time format as the LOOP_INTERVAL

# No website should ever take longer then a second to respond.

TIMEOUT = 1.0

HEADERS = {'user-agent': ('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5)'
                          'AppleWebKit/537.36 (KHTML, like Gecko)'
                          'Chrome/45.0.2454.101 Safari/537.36'),
                          'referer': 'http://stats.nba.com/scores/'}


URL = (
    "https://data.josefwagner.net/intvid_transcreen"
    "/php-functions/active_triggers.php"
)


import socket
import traceback
import requests

RUN_IN_LOOP = eg.event.string == 'Main.OnInit'


def exit():
    eg.PrintError(traceback.format_exc())

    if RUN_IN_LOOP:
        eg.scheduler.AddShortTask(LOOP_INTERVAL , get_request)
      
        eg.plugins.Speech.TextToSpeech(u'IVONA 2 Emma OEM - British English female voice []', -1, u'something is going wrong', u'', 100, u'Lautsprecher (Realtek High Definition Audio)')
            
    else:

        eg.StopMacro()
        eg.Exit()


def get_request():
    try:
        
        
        print "trying request"
        request = requests.get(URL, headers=HEADERS, timeout=TIMEOUT, verify=False)
    except requests.exceptions.Timeout:
        eg.PrintError('-- TiMeOuT ' + ('-' * 25))
        exit()
    except requests.exceptions.TooManyRedirects:
        eg.PrintError('-- ToO mAnY redirects ' + ('-' * 26))
        
        exit()
    except requests.exceptions.RequestException as e:
        eg.PrintError('-- other exception joe ' + ('-' * 20))
        exit()
    else:
        print "request successful"
        todomi = request.text
        if todomi:
            eg.TriggerEvent(prefix="CommandFromDominique", suffix=todomi)
            
    if RUN_IN_LOOP:
        eg.scheduler.AddShortTask(LOOP_INTERVAL, get_request)


get_request()



User avatar
kgschlosser
Site Admin
Posts: 5508
Joined: Fri Jun 05, 2015 5:43 am
Location: Rocky Mountains, Colorado USA

Re: timeout pythonscript after some time?

Post by kgschlosser » Sun Sep 30, 2018 11:25 pm

Yeah. Change your dns servers in your network setup on your PC...


Primary 8.8.8.8
Secondary 8.8.4.4

This can be set seperatly from the IP address of your NIC. So if you are using DHCP that will still function.

Also open a cmd prompt as administrator.

Run this command
Ipconfig /flushdns
If you like the work I have been doing then feel free to Image

zen_SuR
Posts: 11
Joined: Mon Sep 17, 2018 5:12 pm

Re: timeout pythonscript after some time?

Post by zen_SuR » Mon Oct 01, 2018 11:09 am

ok... i am trying that now. so far it looks good.

do you have any background why this could be the solution? without that everything worked also but only after some time it crashed... i spent so much time now with that problem and i want to know everything about it ;)


EDIT:
no... that wasn't it - i changed the dns to yours in my router (did not find anything in my modem-settings) and on my computer on the wifi-adapter... tracrt shows me that 8.8.8.8 is used in cmd also...

still it comes to that point where it starts to timeout... firstly every 10th run, then every 5th run and suddenly no requests go through. i have to stop eventghost. then wait about 10 minutes before it works again for an hour or so... actually i don't even have to restart the modem... just wait...

eventhost gives me the following error each time it times out:

Code: Select all

trying request
-- TiMeOuT -------------------------
Traceback (most recent call last):
  File "0", line 63, in get_request
  File "C:\Program Files (x86)\EventGhost\lib26\site-packages\requests\api.py", line 55, in get
    return request('get', url, **kwargs)
  File "C:\Program Files (x86)\EventGhost\lib26\site-packages\requests\api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Program Files (x86)\EventGhost\lib26\site-packages\requests\sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Program Files (x86)\EventGhost\lib26\site-packages\requests\sessions.py", line 559, in send
    r = adapter.send(request, **kwargs)
  File "C:\Program Files (x86)\EventGhost\lib26\site-packages\requests\adapters.py", line 384, in send
    raise Timeout(e, request=request)
Timeout: (<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x05037A50>, 'Connection to data.josefwagner.net timed out. (connect timeout=1.0)')

some cache is being filled up or some leak-kind-of-thing... any other ideas? i am so horribly stuck here...

EDIT2:
i also considered that the webhoster himself blocks the repeating requests after some time but they looked into their logs and said that every request was answered with "200" and it looks like the failures do not even reach them because they get stuck somewhere before...

i tried to not let it timeout to get some other kind of error... exception gives me this:

Code: Select all

-- other exception joe --------------------
Traceback (most recent call last):
  File "0", line 63, in get_request
  File "C:\Program Files (x86)\EventGhost\lib26\site-packages\requests\api.py", line 55, in get
    return request('get', url, **kwargs)
  File "C:\Program Files (x86)\EventGhost\lib26\site-packages\requests\api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Program Files (x86)\EventGhost\lib26\site-packages\requests\sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Program Files (x86)\EventGhost\lib26\site-packages\requests\sessions.py", line 559, in send
    r = adapter.send(request, **kwargs)
  File "C:\Program Files (x86)\EventGhost\lib26\site-packages\requests\adapters.py", line 375, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPSConnectionPool(host='data.josefwagner.net', port=443): Max retries exceeded with url: /intvid_transcreen/php-functions/active_triggers.php (Caused by <class 'socket.error'>: [Errno 10060] Ein Verbindungsversuch ist fehlgeschlagen, da die Gegenstelle nach einer bestimmten Zeitspanne nicht richtig reagiert hat, oder die hergestellte Verbindung war fehlerhaft, da der verbundene Host nicht reagiert hat)


User avatar
kgschlosser
Site Admin
Posts: 5508
Joined: Fri Jun 05, 2015 5:43 am
Location: Rocky Mountains, Colorado USA

Re: timeout pythonscript after some time?

Post by kgschlosser » Mon Oct 01, 2018 4:53 pm

From what you stated earlier. About having websites that still work and only new ones do not. That is a DNS problem.

When you key in a website that website have to get translated into an up address. Now this gets cached on your local computer and depending on the router there as well.

Now what I had you do was locally. The best place to change those DNS entries are actually in your router. This way your local network will remain the same..

There is a catch here. We do not know where the issue is. It can be in multiple places. We have confirmed that the issue is not local.

Some routers do a DNS passthrough and some forward the requests. If it acts as a pass through the DNS servers given to you by the DHCP on your router will not be the same as the default gateway. If they are the same as the default gateway then it is forwarding. The issue could possibly lie with your router. There are also routing tables stored in your router that could possibly be getting messed up. Or it could be an issue with the node you are connected to from your internet service provider.

I am going to need to know the name of the company as well as the make and model of any routers and cable modems, ISDN modems, DSL whatever it is that gives you the connection to the internet.

Are you using WiFi for your machines connection to your network. And if so. What is the brand / model of the wifi adapter.

If you are running EG on a laptop over WiFi there is a very strong possibility that you have one of the Intel wifi cards that cause an issue very similiar to what you are having. Only fix for that is to buy a new wifi adapter for your laptop.

If you are using an Arris cable modem those things have issues as well. And can cause very similiar issues.

I am going to need to know alittle more about your setup.


Open a cmd prompt as administrator.
Key in this command

Ipconfig /all

Copy and paste the output into a PM to me. The command prompt copy is kind of funky. It's called Mark. Then select the text then press enter and it will copy it.

Or you can do
Ipconfig /all > %USERPROFILE%\Desktop\ipconfig_output.txt


That will make a file on your desktop of the output. Nothing will get displayed I the command window.

Get that information over to me I have had quite a bit of exposure to all sorts of network hardware and have seen all kinds of issues. So something might throw up a flag.

Before you do that change your DNS servers back to their original settings.
If you like the work I have been doing then feel free to Image

User avatar
kgschlosser
Site Admin
Posts: 5508
Joined: Fri Jun 05, 2015 5:43 am
Location: Rocky Mountains, Colorado USA

Re: timeout pythonscript after some time?

Post by kgschlosser » Mon Oct 01, 2018 4:56 pm

Also the error you are getting states that you connected to the server. Sent in your request. But the server did not respond.

Is this the only website you have an issue with??
If you like the work I have been doing then feel free to Image

zen_SuR
Posts: 11
Joined: Mon Sep 17, 2018 5:12 pm

Re: timeout pythonscript after some time?

Post by zen_SuR » Mon Oct 01, 2018 9:50 pm

OK... I FIXED IT.

for real now.

i just phoned with my internetprovider and after some yadayada they changed me from IP6 to IP4 (and also gave me a static IP in the process as a nice side effect - something they always insisted is impossible :-) )

now it is smooth for half a day already.

as you said with your catch - we dont know whats happening and where the mistake occurs... apparently it had something to do with them. i really hope somebody is helped with this thread one day.

and great great thanks to kgschlosser! you are amazing and a big big help to people like me and for a last resort. thank you so much. i learned incredibly much.

greetings!

User avatar
kgschlosser
Site Admin
Posts: 5508
Joined: Fri Jun 05, 2015 5:43 am
Location: Rocky Mountains, Colorado USA

Re: timeout pythonscript after some time?

Post by kgschlosser » Mon Oct 01, 2018 10:30 pm

even if a problem is not caused or has nothing to do with EG I will always help to the best of my ability. I am a problem solver and I do enjoy the responses that i get from a user when everything gets sorted out.

I didn't even think about ipv6 If I had know that is what you were issued then I would have most likely went at that first. even tho it has been around for a long while it still does not work properly the routing from ipv6 to ipv4 and vice versa is crap. it only works sometimes. perfect example right here. I feel pretty confident that you issue will be sorted now also. That is the reason why when you changed the DNS entries nothing happened. But you should still use those numbers I gave you. those are Googles DNS servers. they never go down and they are quick. There isn't an ISP on the planet that has the computing power that google has.
If you like the work I have been doing then feel free to Image

zen_SuR
Posts: 11
Joined: Mon Sep 17, 2018 5:12 pm

Re: timeout pythonscript after some time?

Post by zen_SuR » Tue Oct 02, 2018 6:07 pm

kgschlosser wrote:
Mon Oct 01, 2018 10:30 pm
even if a problem is not caused or has nothing to do with EG I will always help to the best of my ability. I am a problem solver and I do enjoy the responses that i get from a user when everything gets sorted out.

I didn't even think about ipv6 If I had know that is what you were issued then I would have most likely went at that first. even tho it has been around for a long while it still does not work properly the routing from ipv6 to ipv4 and vice versa is crap. it only works sometimes. perfect example right here. I feel pretty confident that you issue will be sorted now also. That is the reason why when you changed the DNS entries nothing happened. But you should still use those numbers I gave you. those are Googles DNS servers. they never go down and they are quick. There isn't an ISP on the planet that has the computing power that google has.
eg really wasn't at fault but the mistake was so hard to reproduce that it felt like voodoo... it never crossed my mind that it could be the ip4>ip6 problem... and i still think somebody f****** up when inventing ip6 :-)

its the first time seeing my eventghost project (more on that very soon...) running for more than 2 days without crashing and i cant believe i got it stable since it is quite complex.... it just feels great and its people like you who make this possible.

i stick to the assigned dns now since they work... but as soon i run into problems i change them back to yours.

thx mate. greetings from vienna, austria :)

Post Reply