Drawing graphs in EG

[HTPCanwender]

I think, I have found a solution. The .subplots method creates a new window on every call and I did not find a way to close the open windows. So I decided to draw the plot in one figure and wipe it after saving the picture. It's like using an chalkboard at school.

There is one issue left. As you suggested I decided to store the data in a more space and time saving format. The time is now in unix format. I did not find a way to transform it back to a human readable format in the graph.

Attached you will find a sample of the input data and the actual script. For testing I reduced the measuring interval to 1 min.

Code: Select all

1574010839.0, 20.9, 46.7, C
1574010899.0, 20.9, 46.7, C
1574010959.0, 20.9, 46.8, C
1574011019.0, 20.9, 46.8, C
1574011079.0, 20.9, 46.8, C
1574011139.0, 20.9, 46.7, C
1574011199.0, 20.9, 46.8, C
1574011259.0, 20.9, 46.8, C
1574011319.0, 20.9, 46.9, C
1574011379.0, 20.9, 46.9, C
1574011439.0, 21.0, 47.0, C
1574011499.0, 20.9, 46.9, C
1574011559.0, 21.0, 46.9, C
1574011619.0, 21.0, 47.0, C
1574011679.0, 21.0, 46.9, C
1574011739.0, 21.0, 46.9, C
1574011799.0, 21.0, 46.9, C
1574011859.0, 21.0, 46.9, C
1574011919.0, 20.9, 46.9, C
1574011979.0, 20.9, 46.9, C
1574012039.0, 21.0, 46.9, C
1574012099.0, 21.0, 46.9, C
1574012159.0, 21.0, 46.9, C
1574012219.0, 21.0, 46.9, C
1574012279.0, 21.0, 46.9, C
1574012339.0, 21.0, 46.9, C
1574012399.0, 21.0, 46.8, C
1574012459.0, 21.1, 46.9, C
1574012519.0, 21.0, 46.8, C
1574012579.0, 21.1, 46.9, C
1574012639.0, 21.1, 46.8, C
1574012699.0, 21.1, 46.8, C
1574012759.0, 21.1, 46.8, C

Code: Select all

import matplotlib.pyplot as plt
import numpy as np

# Zum Glätten der Kurve
# =====================
import sys
sys.path.append('C:\Daten\Programmdaten\EventGhost\Code')
import savitzky_golay as sg

# Ein- und Ausgabepfade:
# ======================
GeraetePfad = 'C:/Daten/Programmdaten/EventGhost/Geraete/' # input
Geraet      = 'Sonoff-33'
Modul       = 'AM2301'
WebPfad     = 'C:/Daten/Programmdaten/EventGhost/web/'     # output

# Daten aus Datei einlesen:
# =========================
Data = np.loadtxt(GeraetePfad + Geraet +'_' + Modul + '.txt', delimiter = ',', usecols = (0,1,2))
x  = Data[:,0]
y1 = Data[:,1]
y2 = Data[:,2]

# Kurve glätten:
# ==============
y1 = sg.savitzky_golay(y1, window_size=27, order=3)
y2 = sg.savitzky_golay(y2, window_size=27, order=3)

# Kurven plotten:
# ===============

Bild = plt.figure(num=1, figsize=(16,9))

ax1 = Bild.add_subplot(1, 1, 1, label='erster Subplot')
ax1.plot(x, y1, label='Temperatur',  color='tab:red',  linestyle='solid', linewidth=5)
ax2 = ax1.twinx() 
ax2.plot(x, y2, label='Luftfeuchte', color='tab:blue', linestyle='solid', linewidth=5)

# Decorations
# ax1 (left Y axis)
ax1.spines['bottom'].set_color('white')
ax1.spines['top'].set_color('white')
ax1.spines['right'].set_color('white')
ax1.spines['left'].set_color('white')
ax1.set_xlabel('Zeit', fontsize=20)
ax1.tick_params(axis='x', rotation=45, labelsize=20)
ax1.set_ylabel('Temperatur [C]', color='tab:red', fontsize=20)
ax1.tick_params(axis='y', rotation=0, labelcolor='tab:red', labelsize=20 )
ax1.grid(alpha=.4)

# ax2 (right Y axis)
ax2.spines['bottom'].set_color('white')
ax2.spines['top'].set_color('white')
ax2.spines['right'].set_color('white')
ax2.spines['left'].set_color('white')
ax2.set_ylabel("Luftfeuchte [%]", color='tab:blue', fontsize=20)
ax2.tick_params(axis='y', labelcolor='tab:blue', labelsize=20)
ax2.set_xticklabels(x[::60], rotation=90, fontdict={'fontsize':10})
ax2.set_title("Temperatur und Luftfeuchte Dachboden (33)", fontsize=22, color='white')
Bild.tight_layout()

plt.savefig(WebPfad + Geraet +'_' + Modul, transparent = True)
ax2.remove()
ax1.remove()

[kgschlosser]

close the buffer...

Code: Select all

buf.close()

[kgschlosser]

you can also ditch the spaces between the items in the data file. they are not needed.
since you are splitting the data file using \n then enumerating the lines in the file and splitting each line by ,

Code: Select all

for line in data.split('\n'):
    date, temperature, humidity, unit = line.split(',')

to convert the unix time into human readable time you want to use

Code: Select all

import time

# this grabs the current unix time you would not use 
# this and is here only for demonstration
# the now would become one of the timestamps from the data file.
now = time.time()

print time.strftime('%c', time.localtime(now))

[HTPCanwender]

I am sorry, but I don't understand the data types for timestamps and how to deal with them while plotting. It's frustrating. I am only able to show at the x-axis the seconds from epoch as stored in the data file. And I am really not a stupid person. Every data conversion I tried lead to error messages. I have red tons of pages on the internet, but there is nowhere an example with a list of timestamps from epoch for use in a plot.

[HTPCanwender]

Finally I found a way to convert the date since epoch stored in the numpy array to a datetime format in a list:

x = dates.num2date(dates.epoch2num(xnp))

The plot shows now a readable time at the x-axis.

2019-11-18 23:35:36+00:00

Until now I did not find a way to format it like

18.11.2019, 23:35:36

All my attempts with the DateFormatter failed.

[HTPCanwender]

I've found the solution. I think, I should be more patient before posting.

[HTPCanwender]

I have one issue left. What do you mean with "close the buffer"? Using plt.close() to close the figure generates the same error message as closing the plot window generated with plt.show() by clicking at the x (close) button. Figure 1 remains in memory all the time as an empty picture and is reused for every new picture.

[kgschlosser]

give this whirly bird a twirl.

I have not tried this at all. it looks right tho..

Code: Select all

# this is a little trick to save a few cpu cycles.
# is seems strange but it works because EG does not delete
# the information in a Python Script after it has run
# so by using the try/except we can test to see if 
# the script has run before. and if it hasn't then 
# do the imports and any constants, no need to 
# create them over and over again

try:
    _ = GeraetePfad
except NameError:
    import numpy as np
    import time
    import os

    # instead of having to add to sys.path and 
    # import a module that is not apart of EG
    # here is an alternative smoothing routine

    def smooth_readings(x_list, y_list):
        poly = np.polyfit(x_list, y_list, 5)
        poly_y = np.poly1d(poly)(x_list)
        return poly_y


    # Ein- und Ausgabepfade:
    # ======================
    GeraetePfad = r'C:\Daten\Programmdaten\EventGhost\Geraete' # input
    Geraet      = 'Sonoff-33'
    Modul       = 'AM2301'
    WebPfad     = r'C:\Daten\Programmdaten\EventGhost\web'  # output

    in_file = os.path.join(GeraetePfad, '{0}_{1}.txt'.format(Geraet, Modul))
    out_file = os.path.join(WebPfad, '{0}_{1}'.format(Geraet, Modul))
    
    # Daten aus Datei einlesen:
    # =========================

    x = []
    y1 = []
    y2 = []
    
import matplotlib.pyplot as plt
global plt

# simple csv file. using numpy to parse it is overkill.
with open(in_file, 'r') as f:
    data = f.read()

for line in data.split('\n'):
    date, reading1, reading2 = list(float(item.strip()) for item in line.split(',')[:3])
    x += [time.strftime('%c', time.localtime(date))]
    y1 += [reading1]
    y2 += [reading2]

# smoothing the readings.
y1 = smooth_readings(x, y1)
y2 = smooth_readings(x, y2)

# Kurven plotten:
# ===============

Bild = plt.figure(num=1, figsize=(16,9))

ax1 = Bild.add_subplot(1, 1, 1, label='erster Subplot')
ax1.plot(x, y1, label='Temperatur',  color='tab:red',  linestyle='solid', linewidth=5)
ax2 = ax1.twinx()
ax2.plot(x, y2, label='Luftfeuchte', color='tab:blue', linestyle='solid', linewidth=5)

# Decorations
# ax1 (left Y axis)
ax1.spines['bottom'].set_color('white')
ax1.spines['top'].set_color('white')
ax1.spines['right'].set_color('white')
ax1.spines['left'].set_color('white')
ax1.set_xlabel('Zeit', fontsize=20)
ax1.tick_params(axis='x', rotation=45, labelsize=20)
ax1.set_ylabel('Temperatur [C]', color='tab:red', fontsize=20)
ax1.tick_params(axis='y', rotation=0, labelcolor='tab:red', labelsize=20 )
ax1.grid(alpha=.4)

# ax2 (right Y axis)
ax2.spines['bottom'].set_color('white')
ax2.spines['top'].set_color('white')
ax2.spines['right'].set_color('white')
ax2.spines['left'].set_color('white')
ax2.set_ylabel("Luftfeuchte [%]", color='tab:blue', fontsize=20)
ax2.tick_params(axis='y', labelcolor='tab:blue', labelsize=20)
ax2.set_xticklabels(x[::60], rotation=90, fontdict={'fontsize':10})
ax2.set_title("Temperatur und Luftfeuchte Dachboden (33)", fontsize=22, color='white')
Bild.tight_layout()

plt.savefig(out_file, transparent=True)
ax2.remove()
ax1.remove()

del ax1
del ax2
del Bild

def do():
    global plt
    plt.close()
    del plt

wx.CallAfter(do)

# this only deletes the contents not the actual variable
del x[:]
del y1[:]
del y2[:]

[HTPCanwender]

Before I test it, one question please:

try:
_ = GeraetePfad

Has the variable GeraetePfad to be unique in my EventGhost universe? I am using it as local variable also in other scripts. If I need a unique idintifyer, perhaps I should choose an other variable or define a special one for the script identification.

[kgschlosser]

the scripts get saved in their own namespace. so they are not global. again the key word to that is "saved" meaning that once the script gets run for the first time the variables get created and stored. so when it runs a second time the variables already exist. so there is no need to have the program create constants over and over again.

as a tutorial so you can see what is happening paste the code below into a python script and run it.

Code: Select all

try:
    some_varaible += 10
except NameError:
    some_variable = 10
    
print some_variable    

you are going to get "10" printed to your log.
now run it again..

it's not "10" that gets printed


that is the reason why it is important when dealing with large data structures you delete them at the end of a python script because they are going to consume memory.

the only case where you do not delete it is if you continue to add data to the structure and use it. like in the above example.

[HTPCanwender]

First the very good news. The way you suggested to close the plot is stable and works perfect. Here is the actual version of the script. Maybe it can be the blueprint for someone else.

Code: Select all

import matplotlib.pyplot as plt
global plt
import numpy as np
import matplotlib.dates as mdates

# Ein- und Ausgabepfade:
# ======================
GeraetePfad = 'C:/Daten/Programmdaten/EventGhost/Geraete/' # input
Geraet      = 'Sonoff-33'
Modul       = 'AM2301'
WebPfad     = 'C:/Daten/Programmdaten/EventGhost/web/'     # output

# Daten aus Datei einlesen:
# =========================
Data = np.loadtxt(GeraetePfad + Geraet +'_' + Modul + '.txt', delimiter = ',', usecols = (0,1,2))
xnp  = Data[:,0]
y1   = Data[:,1]
y2   = Data[:,2]

x = mdates.num2date(mdates.epoch2num(xnp))  # Zeitstempel von Unix-Zeit in datetime-Zeit umwandeln

# Kurven plotten:
# ===============
Bild = plt.figure(num=1, figsize=(16,9))
ax1  = Bild.add_subplot(1, 1, 1, label='erster Subplot')
ax1.plot_date(x, y1, xdate=True, label='Temperatur',  color='tab:red',  linestyle='solid', linewidth=2)
ax2  = ax1.twinx() 
ax2.plot_date(x, y2, xdate=True, label='Luftfeuchte', color='tab:blue', linestyle='solid', linewidth=2)

# Decorations
# ax1 (left Y axis)
ax1.spines['bottom'].set_color('white')
ax1.spines['top'].set_color('white')
ax1.spines['right'].set_color('white')
ax1.spines['left'].set_color('white')
ax1.set_xlabel('Zeit', fontsize=20)
ax1.tick_params(axis='x', rotation=45, labelsize=20, colors='white', labelcolor='black')
ax1.set_ylabel('Temperatur [C]', color='tab:red', fontsize=20)
ax1.tick_params(axis='y', rotation=0, labelcolor='tab:red', labelsize=20, colors='white' )
ax1.grid(alpha=.4)
ax1.xaxis.set_major_formatter(mdates.DateFormatter("%d.%m.%Y, %H:%M"))

# ax2 (right Y axis)
ax2.spines['bottom'].set_color('white')
ax2.spines['top'].set_color('white')
ax2.spines['right'].set_color('white')
ax2.spines['left'].set_color('white')
ax2.set_ylabel("Luftfeuchte [%]", color='tab:blue', fontsize=20)
ax2.tick_params(axis='y', labelcolor='tab:blue', labelsize=20, colors='white')
ax2.set_xticklabels(x[::60], rotation=90, fontdict={'fontsize':10})
ax2.set_title("Temperatur und Luftfeuchte Dachboden (33)", fontsize=22, color='white')
ax2.xaxis.set_major_formatter(mdates.DateFormatter("%d.%m.%Y, %H:%M"))
Bild.tight_layout()

plt.savefig(WebPfad + Geraet +'_' + Modul, transparent = True)
ax2.remove()
ax1.remove()

del ax1
del ax2
del Bild

def do():
    global plt
    plt.close()
    del plt
    
wx.CallAfter(do)

Now it's time to optimize the code with respect to the calculation time. I will try your further proposals during the next days. Until now I am not shure if I want to say goodbye to the numpy arrays. They might be useful for further data processing.

I found one issue yet. The code

Code: Select all

for line in data.split('\n'):
    date, reading1, reading2 = list(float(item.strip()) for item in line.split(',')[:3])
    x  += [time.strftime('%c', time.localtime(date))]
    y1 += [reading1]
    y2 += [reading2]

leads to an error after processing the last line of the file. The last line of my data file (like every line) ends with "\n". The .spilt("\n") produces therefore a list with an empty last element. I corrected it by adding .pop(). I hope, this is a suitable solution.

Code: Select all

for line in data.split('\n').pop():
    date, reading1, reading2 = list(float(item.strip()) for item in line.split(',')[:3])
    x  += [time.strftime('%c', time.localtime(date))]
    y1 += [reading1]
    y2 += [reading2]

[kgschlosser]

That error is becaus of the trailing new line. All you have to do is add into the iteration

Code: Select all

for line in data.split('\n'):
    if not line.strip():
        continue

[kgschlosser]

Using pop() is going to be slower processing. pop() removes the line from the list and returns it. We really do not need to have the line removed and this takes extra time to do. It will be noticable on a large amount of data.

[HTPCanwender]

Now I have an other issue. My smart home computer is tiny, low power consuming and slow. After collecting data for more than one week the calculation time for one plot takes 4 sec. During this time the triangle in the status bar turns red. During this time EG seems to be blocked for other tasks. Now my question. Is it possible to run the script for plotting the figure in the background so that EG is not blocked?

[kgschlosser]

OK so we are going to want to run the creation of the plots in it's own thread. so it doesn't stall EG. we will use this same thread as a timer for when to build the graph.

Here is the basic outline. You need to add your current script to this script. This script is going to get put into your autostart group. move it to the bottom of the autostart group.
The commented lines that are in capitals are where you are going to have to add in code from the original script.

This script only needs to be run a single time. It is going to exit the thread when you close EG. It has a wait timer already built into it. I set the timer to generate a graph every 5 minutes. you can change this by adjusting the WAIT_DURATION variable.

I set the script up so that if an error occurs it will print out the error and exit the thread. you will be able to run the script again after you make changes to fix the error. you can also close the thread manually by running eg.globals.shutdown_plot_thread() from another python script. otherwise you will need to shutdown EG and restart it.

the script will not run more then a single time if the thread is running. we do not want to have more then one thread accessing the same files.

Code: Select all

try:
    if event.is_set():
        event.clear()
    else:
        eg.PrintError('The thread is currently running')
        eg.Exit()
except NameError:
    import threading
       
    event = threading.Event()
    
    def on_eg_close(evt=None):
        if evt is not None:
            eg.Unbind('Main.OnClose', on_eg_close)
        
        if t.is_alive():
            event.set()
            t.join()
            
        return False
    
    eg.globals.shutdown_plot_thread = on_eg_close

    eg.Bind('Main.OnClose', on_eg_close)
    
    # IF YOU HAVE THE TRY/EXCEPT AT THE TOP OF YOUR CURRENT SCRIPT
    # ADD THE PARTS FROM THE EXCEPT HERE

WAIT_DURATION = 300 # seconds 300 = 5 minutes

def loop():
    # this is to stall the creation of the first
    # graph. we want to make sure that EG is fully started up
    # before running the plot.
    event.wait(20)
    while not event.is_set():
        try:
            pass
            # DELETE THE LINE ABOVE THIS AND PLACE ALL PLOTTING CODE HERE
        except:
            import traceback
            traceback.print_exc()
            event.set()

        event.wait(WAIT_DURATION)

t = threading.Thread(target=loop)
t.start()