Added database feature.

[RaBa64]

The database ist stored persistently on LittleFS.
The AC total energy is written every hour to the database, together with a timestamp. Each entry to the database requires 8 bytes on the LittleFS partition. The database can be read with the API call /api/database

ToDo:
UI component is still missing.
Modification for chunked response needed.

Ralf Bauer

P.S.: Thanks for the great job on OpenDTU !

[ituri]

Looking excellent. Really hoping for this to get merged.🙏

Btw, your inverter's serial bumper is visible in the screenshot. You might want to edit that out.

[RaBa64]

I have one last problem with the code.
I get an error when building the distribution files with "yarn build".
It says that it cannot find the google library.
Everything works fine with "yarn dev".
Maybe somebody can help me with that (it is my first Vue project).

[RaBa64]

Looking excellent. Really hoping for this to get merged.🙏

Btw, your inverter's serial bumper is visible in the screenshot. You might want to edit that out.

Thank you for the notification about the serial number.
I will upload a new screenshot.

[tbnobody]

Just a few first remarks:

• Do not include libraries from external servers (maybe the google chart lib has to be replaced by something more lightweight, don't now how much space it requires now)
• You placed the chart in the inverter section of the live view. This is very confusing when having multiple inverters
• You edited parts of the vue code without changing any logic in this code sections
• Each hour consumes 9 byte of data. That means ~80kb / year. The size of the data partition is ~200kb. LittleFS is using copy on write. That means after ~1,5 years you cannot any additional data.
• Due to the copy on write behavior the writing duration of writing the data will increase over time. Also the wear out of the flash will increase over time as the whole file will be rewritten.

[RaBa64]

Thank you for your comments, let me clarify a few points.

• The Google Chart lib does not require any space on the ESP32, except for the Vue wrapper. The library is directly loaded and run by the browser. I also want to implement a calendar chart to show the enery for each day.
• Yes this is a bug, the chart must be outside the inverter section. I'll fix that.
• I'm working with Vue.js for 2 days now, so please be patient with me ;-)
• Yes its 8 bytes per hour with energy generation. So there will be no data stored during night hours. In summer this makes around 15-16 data points, in winter it will be much less. Calculating with an average of 10 data points per day, it will be 3650 data points or 29.200 bytes per year. So 200kB is enough for almost 7 years. It would also be possible to implement a purge of old data. There should also be no excessive wear on the FLASH. Calculating with 100.000 write cycles, the FLASH should at least last for 27 years, without taking the wear leveling into account.
• It is also possible to keep all data points for one day in RAM and write them only once a day to the filesystem. As long as you don't unplug the power supply this will work fine. All you would loose is one day of data.

[tbnobody]

• The Google Chart lib does not require any space on the ESP32, except for the Vue wrapper.

So it will not work offline or in AP mode. And you are transfering data to google in US (which will require a DSGVO hint and a Opt-Out)..... As I said... please no libraries from external servers....

So 200kB is enough for almost 7 years.

But you cannot extend a 150kb by 10bytes if you have only 50kb left... As I said, LittleFS is a copy-on-write filesystem. That means at each file operation, it creates a copy of the file, makes the changes and adjusts the file pointer in the last step to be 100% sure that there is no corrupted file system.

[RaBa64]

• Only the Google Chart library code (JS) is loaded in the browser. The rendering of the charts is done in the browser without sending any data to Google servers. But I agree, this feature can only be used online.
• You are right, LittleFS is a COW filesystem. But fortunately it is implemented very intelligently. If you append data to a file, it does not copy the already existing data (see here).

[tbnobody]

Only the Google Chart library code (JS) is loaded in the browser.

Thats enough to require a DSVGO hint because the IP address of the fetch operation is transmitted to google.

[t2ton]

Hi guys,
I think 1 hour database is not really usefull. Even for possible troubleshooting of the system it is way too rough and slow. In same way one could just saved a day total close to midnight time..
I would really like (as many others) an SD card offline logging feature. It shouldn't be hard to code. Simple saving "time: power" feature. Probably with option in GUI on time step. way more data, but who cares - SD capacities are huge nowadays. I personally would like to have a shortest step (5s?), as it may be used afterwards for our PhD students on simulation of solar light utilization in different off-grid systems.
One can of course do one another ESP32 which will just ask for data and save, but it is somehow an overkill - one has one MC already, which works 24/7 right?
So it would be great to have SD feature. Or you just dont want to steal the cookies from Ahoi guy? :)

[RaBa64]

@t2on, I have also thought about stoing the data on an SD-card, but I decided to stay with my current solution and store all high resolution data in my InfluxDB, which is much easier and faster to query. This is pretty easy to do by using MQTT and Telegraf.
I use the 25 hours chart in OpenDTU only for a brief overview of the energy production.
What is still on my todo list is to integrate a calendar chart, besides the bar chart, to have also an overview on a day by day basis for all years, similar as the one on the original Hoymiles website / App.

[CommanderRedYT]

Imo NVS partition would be better fitted than LittleFS

[pkese]

How about just using an external Google spreadsheet for storage:

• user creates a spreadsheet on Google and enters url into OpenDTU (it's an opt-in for user)
• OpenDTU then knows that user is fine with acessing Google services including external Google Charts javascript on web page
• bonus feature: user gets a log of inverter statistics on Google Sheets that they can use for other purposes

[stefan123t]

@pkese yes this would be the next best thing to do, if not for the privacy concerns involved.
I already logged an issue nextcloud/server#40968 in order to add SabreDAV http-PATCH feature to nextcloud/owncloud too.

Currently appending to files via WebDAV is neither announced nor supported by the backend.
Though I would assume appending is supported on local Filesystems and most likely also supported by other storage backends.
The second part of the feature nextcloud/server#15923 for updating some random/middle parts of a file would not be required for our basic purpose.

[CommanderRedYT]

@pkese But how is this accessible while being offline?

Also, I think the Prometheus API would be better fitted