Chrome Extension - Browser Events

Questions and comments specific to a particular plugin should go here.
dan Edens
Experienced User
Posts: 98
Joined: Mon Sep 24, 2018 7:57 pm

Re: Chrome Extension - Browser Events

Post by dan Edens » Sat Nov 23, 2019 10:54 pm

This topic is high on my priorities. but back to the extension for a sec.

Just a update after several projects using it.

I love the fact the event fires after the page is finished loading, This is vastly superior to using chrome tab name change parsing.
And i love that it re-fires on a tab switch, that was smart. Having a value for "is active tab" when it finishes loading would help alot.
This has given me problems with auto refreshing pages, if chrome crashes and restores them all, or when opening groups of tabs.

Otherwise this thing has been working liek a charm. I have given up on moving the mouse for clicks, in favor of sending keystrokes and using shit like

Code: Select all

wait(0.2)
key({ctrl+f}{buttontext})
wait(0.1)
key(esc)
wait(0.1)
key(enter)

Medy
Experienced User
Posts: 93
Joined: Sat Nov 15, 2014 8:33 pm
Location: Germany

Re: Chrome Extension - Browser Events

Post by Medy » Sun Nov 24, 2019 12:21 am

yes will see if I can add a check of active tab .

Since I also dont like all the events firing when chrome loads ...

I was monitoring it ... but it seems usless to keep it in :D...

When the next extension comes I thought about
a action to query all open tabs / pinned / unpinned , as well as a action to close tabs.

so you could maybe autoclose tabs with a filter of your choice ... and repopulate at will by creating new tabs :D ...
I sometimes have too many tabs open .. so that would be a cool cleanup automation xD .


Atm kgschlosser is helping me getting a basic plugin setup going with websockets.

Once that is running ... I can add add a bunsh of stuff to the extension.
Since then It will have 2 way commincation ... I want to remove the need of having to create
makros like the ones you are using .

With 2 way dataflow the plugin can just pass in the values for the extension to nativiley open/ close tabs.

dan Edens
Experienced User
Posts: 98
Joined: Mon Sep 24, 2018 7:57 pm

Re: Chrome Extension - Browser Events

Post by dan Edens » Sun Nov 24, 2019 12:45 am

Yeah it would be REALLY cool to be able to eat and spit tabs that have been idle or something.
I use the "great suspender" tab to kinda manage, but just isn't... right, I'd much rather be able to play with it in EG.
Would be really cool to get the virtual desktops plugin working and be able to target tabs to diffrent windows on diffrent desktops.
image.jpg

Another neat one would be to find a way to Send keystrokes to an inactive tab. For instance to keep a page from idling and timing out.
There are probably already extension for that and I'm being lazy.

Really cool tho I'm excited to see where it goes.

Medy
Experienced User
Posts: 93
Joined: Sat Nov 15, 2014 8:33 pm
Location: Germany

Re: Chrome Extension - Browser Events

Post by Medy » Mon Nov 25, 2019 12:09 am

So to not getting logged out of websites ?

I dont want to put a big amount of features into this plugin ... there will be maybe about 10 different actions in total .....

you could kinda combine them and do all sorts of stuff ;) ... the datasets and actions will allow for that. That is the goal


There will be
Tab Actions
Page actions
and maybe Notification actions (I was thinking it would be neet to be able to react to other extensions notification & or create Notification yourself) maybe with clickable links and stuff.
Like finally you are getting notifications in the browser you actually want xD

I think im almost done Brainstorming about this .. since I have a working foundation I will get to it soon to get the first version up... with only a small number of features .... I want to montior how it behaves before I add tons fo stuff to it :D..


Maybe I will try getting aproved in the appstore ocne more ... apperently last time my description was too vague xD ... I need to be suepr transparent for their process

dan Edens
Experienced User
Posts: 98
Joined: Mon Sep 24, 2018 7:57 pm

Re: Chrome Extension - Browser Events

Post by dan Edens » Mon Nov 25, 2019 12:19 am

Actually i thought it through a little bit, All that would mean would be the ablity to change the tab.
If i wanted to simplify it even more, Just retrieving the order of tabs would be enough to do it.
EG already has the send keystroke bits.

Yeah nvm I'll just stay tuned haha.

User avatar
kgschlosser
Site Admin
Posts: 5206
Joined: Fri Jun 05, 2015 5:43 am
Location: Rocky Mountains, Colorado USA

Re: Chrome Extension - Browser Events

Post by kgschlosser » Mon Nov 25, 2019 2:33 am

have a look at this.

https://developer.chrome.com/extensions ... scripts#pi

using this we can pass the HTML document to EG and EG can then pass back a generated javascript that registers for notifications for the various events that are available.

so we would be able to generate events for things like mouse clicks on the various elements. or we can also set values for elements this is the key piece that is needed to interact with a website from EG.

opening and manipulating the tabs and the general UI of chrome is also a good thing. but having the ability to interact with the page that is where it's at.
If you like the work I have been doing then feel free to Image

Medy
Experienced User
Posts: 93
Joined: Sat Nov 15, 2014 8:33 pm
Location: Germany

Re: Chrome Extension - Browser Events

Post by Medy » Sun Dec 01, 2019 7:51 am

Yes Im planning on adding Page functionality eventually.

...
- interact with pages
- Browser Notifications
- manage Downloads
- Bookmarks

I do it little by little:


I have the new plugin now at the level of the previous one.
(previous one used webserver) and had only 1 way communication.


Seems to be working so far... doing some testing...
Will add features for moving tabs / get tab position next ;) ...
That being said im new to python so its going slow xD... probably next weekend or so.

if you like you can explain your use-case for tabs .. for what you have in mind.

Moving tabs is possible when
- creating tabs
- after they are created

Code: Select all

{"command":"ActiveTab","data":{"active":true,"audible":false,"autoDiscardable":true,"discarded":false,"favIconUrl":"https://s.ytimg.com/yts/img/favicon_32-vflOogEID.png","height":648,"highlighted":true,"id":330,"incognito":false,"index":0,"mutedInfo":{"muted":false},"pinned":true,"selected":true,"status":"complete","title":"My NEW Editing Setup! LG 65\" OLED C9 TV - YouTube","url":"https://www.youtube.com/watch?v=Ww8havSLa9E","width":1280,"windowId":329}}
I read some more into some of the properties. the discarded value is basicly the "idle" status ...
so I guess you could use that to close tabs once they are idle.

throw me some ideas for commands you would like to have regarding tabs

User avatar
kgschlosser
Site Admin
Posts: 5206
Joined: Fri Jun 05, 2015 5:43 am
Location: Rocky Mountains, Colorado USA

Re: Chrome Extension - Browser Events

Post by kgschlosser » Sun Dec 01, 2019 8:48 am

Medy wrote:
Sun Dec 01, 2019 7:51 am
if you like you can explain your use-case for tabs .. for what you have in mind.
I specifically do not have a use case for the tabs.


the only thing I would do is set id's for the physical tabs themselves. where tab id 0 is the active tab and then number the tabs 1, 2, 3, 4, 5, etc... from left to right. This makes it easy when wanting to direct commands to either the active tab (which can be any id at any time) or to a specific tab..

so say I want to make tab 5 active.. i would use a command like so.

Code: Select all

{
    'tabnum'=5,
    'command'='activate'
}
or if I wanted to move the active tab (say it is number 3 for this example) instead of having to issue 2 commands

sent to extension

Code: Select all

{
    'command': 'get_active_tab_id'
}
sent from extension

Code: Select all

{
    'result': 3
    'response': 'get_active_tab_id'
}
sent to extension

Code: Select all

{
    'tabnum': 3,
    'command': 'move',
    'target': 1
}
sent from extension

Code: Select all

{
    'response': 'move',
    'result': 'success'
}
it becomes simplified into a single transaction

sent to extension

Code: Select all

{
    'tabnum': 0,
    'command': 'move',
    'target': 1
}

sent from extension

Code: Select all

{
    'response': 'move',
    'result': 'success'
}
you could use the the following to get the active tabnumber

sent to extension

Code: Select all

{
    'tabnum': 0,
    'command': 'get_id',
}
sent from extension

Code: Select all

{
    'result': 3,
    'response': 'get_id'
}

I do not know how your chrome extension works. or if it is even possible to have multiple threads. if it is and there is the possibility of performing some kind of a command that may take more then a second to complete you might want to wrestle with the idea of using callback id's so a response can be married to the command sent into the extension.
If you like the work I have been doing then feel free to Image

Medy
Experienced User
Posts: 93
Joined: Sat Nov 15, 2014 8:33 pm
Location: Germany

Re: Chrome Extension - Browser Events

Post by Medy » Sun Dec 01, 2019 11:06 am

Okay so 1 command for jumping to specified tas.
Another one for moving tabs around like this :

Command :

Code: Select all

{
    'tabindex': 3,   
    'command': 'move',
    'targetindex': 1
}

Do you need a response for moving tabs ?
I dont see how itwill be useful , at least for what I had in mind at this point.

Would just assume that any command is a succes,

Cant really tell at this point what you mean with multiple threads ... like you want there to be
some kind of / simultaneous dataflow? .

For now, I was simply thinking of sending commands back and forth ... 1 by 1

User avatar
kgschlosser
Site Admin
Posts: 5206
Joined: Fri Jun 05, 2015 5:43 am
Location: Rocky Mountains, Colorado USA

Re: Chrome Extension - Browser Events

Post by kgschlosser » Sun Dec 01, 2019 12:15 pm

what I mean by multiple threads is if your extension runs in a single thread and there is something that it is doing that may take a while to do it would cause the extension to stop until it was finished. so if it took say 60 seconds to complete something then any commands sent during that 60 seconds would not get processed until after the 60 seconds is up... this would cause a lag. so if the extension has the ability to carry out simultaneous commands then theoretically they may not get processed in the order in which they were intended to depending on how long the execution time takes. using callback id's is a mechanism that can sync this process to ensure that things get done in the order they are intended. and the id that gets sent into the extension also gets returned with a response so there is knowledge of that command completing. It provides a mechanism of being able to chain commands together kind of like what EG does with the macros.

There should always be a response. either successful or failure. an example would be with the tabs.. if there is only a single tab and the user sends in the command to move it to a slot that does not exist there should be some kind of an indicator back letting the user know that an error has occurred. The "no news is good news" Is not one of the best types o communications. take a look at broadcasting using UDP. it is not done all that much. because you simply do not know if the message got to where it was supposed to go.

say an error happens in your extension and the extension gets all goofed up.. so I send a command in and then work under the assumption the extension actually did what it was supposed to do.. and as it turns out it really didn't do what it was supposed to do... so no answer means that something critical happened and the extension is no longer able to respond.. try to sever the connection and see if that give it a bump start.. and if not then we can generate an event about the error. the user can create a macro from that event that kills the process and restarts chrome. and everything is back in business. because chrome is going to register the killed process as an application crash the watchdog will restart it or if the watchdog is not running then EG can restart it, the browser will bring back up the last pages loaded.. So nothing lost in the process but a little bit of time.

This is also a nice mechanism to be able to determine if chrome has flown the coup and to restart it.. Chrome does have some memory management issues and it does lock up..
If you like the work I have been doing then feel free to Image

Medy
Experienced User
Posts: 93
Joined: Sat Nov 15, 2014 8:33 pm
Location: Germany

Re: Chrome Extension - Browser Events

Post by Medy » Mon Dec 02, 2019 2:38 am

Okay that makes sense.

I see how a response will be useful in tat case .. But at this stage & my usecases it didnt cross my mind.

Im not thinking that far ahead ;)....

The extension can probably handle multiple request at once and return whenever the operation is finished.
I think that chrome is built that way.

So yea having identifiers will probably be needed... for what you described.

For everything that isnt visible ... like tab creating ... you need a response for wrtinng scripts

Code: Select all

{
tabindex : 3
identifier:xyz123
action: move
}

Code: Select all

{
response : succes
identifier:xyz123
action: move
}
should these identifiers be manually set by the user ?
Or should I use some library to generate a random id ?
Or I could just use a regular interger 1 , 2 , 3 ..

I would probably choose the last one ...
Guess one could just pass in different value to the action when doing a python script ... if that would be fine
Last edited by Medy on Mon Dec 02, 2019 3:04 am, edited 2 times in total.

User avatar
kgschlosser
Site Admin
Posts: 5206
Joined: Fri Jun 05, 2015 5:43 am
Location: Rocky Mountains, Colorado USA

Re: Chrome Extension - Browser Events

Post by kgschlosser » Mon Dec 02, 2019 3:03 am

the identifier should be something like a GUID. the possibilities of generating 2 ID's that are the same is really large.

This is something done on the EG plugin side of things. there is no need to have any user input to handle this. The GUID gets created and sent with each command. and when that task is finished on the extension side of things that same GUID that was passed then gets returned in the response. This is how the EG Plugin is going to know a task as completed.



this is pseudo code. only for example purposes.

Code: Select all


import uuid
import json

guid = uuid.uuid4()

command = {
    "func_id": guid,
    "command": "new_tab",
    "make_active": true
}
websocket.send(command)

response = websocket.recv()

response = json.loads(response)
if 'func_id' in response and response['func_id'] == guid:
    print 'command completed'

It is much easier to plan things out then to have to go back and change code that is already written. I do not know how much you are going to be able to interact with the browser and the webpage. If what I was reading is correct about being able to inject javascript code to be run in parallel with a website so that the code can interact with the website then the sky is the limit on what can be done as far as programmatically interacting with a website. If that is indeed the case about the javascript all kinds of events can be and actions can become available. Like generating an event in EG if you pay or pause a video on youtube. Or even netflix. Hulu is another.. These are HUGE things to have available because there is nothing currently available to interact in this manner.

If you put together your extension and plugin and it has the ability to generate events when media plays pauses and stops add that to the ZWave plugin for EG and you now have the ability to dim the lights when a movie starts to play, and raise them when it is paused or stopped.
If you like the work I have been doing then feel free to Image

Medy
Experienced User
Posts: 93
Joined: Sat Nov 15, 2014 8:33 pm
Location: Germany

Re: Chrome Extension - Browser Events

Post by Medy » Mon Dec 02, 2019 3:12 am

yes fun fact I wrote a extension for youtube fullscreen events.

Dimming my light when in fullscreen mode ;)...

it not directly using the button events .... but the window object (size widht) ... that way you have clear on and off event

and not just a toggle ;)...


I can add this on the todo list right after moving tabs ;) (injecting scripts)

User avatar
kgschlosser
Site Admin
Posts: 5206
Joined: Fri Jun 05, 2015 5:43 am
Location: Rocky Mountains, Colorado USA

Re: Chrome Extension - Browser Events

Post by kgschlosser » Mon Dec 02, 2019 6:14 am

capturing changes to a media stream in a browser is something I think is one of the "high priority" and most wanted features. If you can also add the ability to control the media stream It would be a pretty big hit I think.


I am saying media stream because if it can work universally across any website that streams any kind of media that would be the thing to have. This way it would work with almost anyone's favorite streaming website.
If you like the work I have been doing then feel free to Image

Medy
Experienced User
Posts: 93
Joined: Sat Nov 15, 2014 8:33 pm
Location: Germany

Re: Chrome Extension - Browser Events

Post by Medy » Mon Dec 02, 2019 10:05 am

maybe some events can be captured / controlled universally .. I would need to test that
https://developer.mozilla.org/en-US/doc ... dia_events

Post Reply