InfluxDB causes extreme CPU load

[Alpha]

Hi,

I am on Mycodo 8.15.12 and everything worked fine after initial installation issues. Interface was always quite slow though. All of a sudden, without changing anything, my RasPi 3 A is basically not responding anymore due to InfluxDB process utizilizing the whole CPU power. As soon as I stop the influxdb process, Mycodo is responding again and is also much more responive than initially.
Anyone who knows this problem? Seems to have been a problem fromm the start which deteriorated over time.There wasn’t much data in the database…
Thanks.

[Lucid3y3 (Jason C LaFleur)]

Yup, been seeing the same excruciating slowdown on both a Pi 3B+ and a Pi4 using InfluxDB v2. The web interface has become so slow it’s practically non-responsive, it can take upwards of 2 minutes for any settings saves or function activation/deactivation to happen, even longer to fully load a dashboard.
Even re-creating the database with a fresh empty file doesn’t seem to help.
Trying to open more than one Mycodo web page causes both pages to hang and never load.
No difference in speed between running the OS and Mycodo from an SD card or a USB drive either.

Using htop from the CLI shows all 4 cores constantly pegged around 65% usage and memory almost all used. All of the files with the most CPU usage are “/usr/bin/influxd”.
And this is after disabling all but a few essential Inputs and Functions to keep my hydroponics system running.
Something has definitely changed recently with Influx, or one of the many other dependencies and packages Mycodo uses… because I’ve been using Mycodo for over 2 years, I haven’t changed my setup, and I’ve never had these problems until the past 7 months.

[Alpha]

I’ve reinstalled Mycodo and it worked very responsive for a few days. Now it is unfortunately unusable again. All it had to process was one temperature sensor input with a measurement every 30s or so.

[Lucid3y3 (Jason C LaFleur)]

Well, the Pi 3 and the Pi 3B+ are actually 64-bit SOC.
Which OS are you using… 32 or 64 bit?
If you are running the 32-bit OS, you need to use the InfluxDB V1.
If you are using the 64-bit OS, you might want to run the InfluxDB V2.

Also, on the Pi 3 models, the WiFi and the SD card share the same I/O bus, so it should noticeably improve performance if you disable the WiFi on the Pi in the config.txt file and use the Ethernet connection to hard-wire the Pi to your router.

dtoverlay=disable-wifi

Make sure you are using a quality brand-name SD card with at least a U3 class speed rating, the faster the better.
The speed of the SD card can greatly effect the overall I/O speed of the Pi.

If you are sure you are using a good ultra-speed SD card, and you are using a Raspberry Pi 3 or 3B+, then you can overclock the SD card slot by adding this line to the end of your /boot/config.txt file…

dtparam=sd_overclock=100

This will set the Pi’s SD card reader to it’s maximum speed to be compatible with the ultra-speed SD card.
It may offer a little improvement in performance, but a Pi 3A is really not much better than the Pi 2. I really recommend a Pi4… or even a new Pi 5 maybe.

[KyleGabriel]

The latest unreleased version of Mycodo upgrades influxdb 2.x to the latest version. You can upgrade to the master branch and see if this fixes your issue.

[Alpha]

Thanks for your replies. Unfortunately I’ve installed a 32 bit OS. Therefore it seems I cannot use InfluxDB V2. Disabling WiFi is no option, since the Pi doesn’t have an Ethernet port. The SD card I use only has a U1 classification.
I’ll try with a 64 bit OS and reinstall everything on a U3 card. I hope it will be resolved by using InfluxDB V2.

[Alpha]

Hi again,

I bought a new high speed U3 class SD card and installed RasPi OS Lite 64 bit and Mycodo from scratch. Unfortunately nothing changed. The system ran great for about a day, now it’s back to the initial issue. Behaviour thefore just like before on Influx v1.
I currently don’t have access to the terminal. However, from what I understand a load average of about 6 queued processes is not good and likely the reason the system is responding extremely slow.

I am running Mycodo 8.15.13 and a Mosquitto server on that machine - nothing more. Mycodo is set to use InfluxDB v2 and processed measurement data of a few MQTT subscriptions. Measurements are maybe a couple of houndred numbers. With the latest 8.15.13 release, the influxdb should be updated to the latest version according to the changelog.
I seem to have something in my specific configuration which causes the system to fail. But given the little functions I use, I have no idea what it may be
Any help very appreciated, thanks.

[Lucid3y3 (Jason C LaFleur)]

At this point, the Pi 3A is a pretty “old” piece of hardware (2018), and it has pretty limited I/O options… only 1 USB-2 port, no on-board Ethernet, and only 512MB of RAM.

The only thing you can do at this point is buy a better Pi… like a Pi 4 or Pi 5. I’d recommend one with 8GB of RAM.

The Pi 4 has on-board USB-3 which is fast enough for connecting an external SSD drive.

The Pi 5 has on-board PCIe 2.0 which would be ideal for connecting an external SSD.

Improving read/write I/O speeds is the only real way to get faster performance from a Raspberry Pi… and that is what the main problem is here… lack of an I/O bus that can keep up with all the sensor and database read/write cycles. I think the other issue here is that there have been changes made to the Raspberry Pi OS in anticipation of the Pi 5 release. Unfortunately, I think these recent changes may have lead to a decrease in performance of the older Pi models when running the latest PI OS.

As soon as I have some extra cash, I plan on upgrading to a Pi 5 and a NVMe SSD drive.

[Alpha]

Fair enough. Might be worth a try. I’m a little worried tho, that it will just be the same with a more powerful Pi. I mean, I assume other people use Mycodo with dozens of Inputs, Outputs, Functions, Controllers etc. also on Pi’s, no? All I did was importing a few measurements (like four data points a minute) and nothing else. It’s hard to imagine for me that it could overburden the machine already.
What can this old Pi do then?
But yeah, happy to spend the cash if it works then.
Thanks.

[KyleGabriel]

This issue appears in a very low minority of installs. None of my dozen or so installs of Mycodo suffer from this issue, and range from Pi 3s to Pi 5s. I do tend to use exclusively SSDs these days because they are so cheap and provide the best reliability and performance boost when compared to using a micro SD. I’d suggest first try using an SSD (with USB to SATA adapter) before upgrading to a new Pi. But I can say the Pi 5 is significantly faster than the Pi 4, and the Pi 4 is significantly faster than the Pi 3.

[Lucid3y3 (Jason C LaFleur)]

UPDATE:

So I recently moved Mydoco from the Pi 3B+ I was running it on to my Pi 4 8gb.
I spent some more time researching SD cards and bought one of these…

https://www.amazon.com/SanDisk-Extreme-microSDTM-Adapter-SDSQXCU-064G-GN6MA/dp/B09X7BYSFG/ref=sr_1_17?crid=1QWJ0BZKQT2IK&dib=eyJ2IjoiMSJ9.t14xbQTFzn97KXs0Z2YRwT5Yb-CjmOtFo6h6KlarJQVUf4RKAFZL_hf1LSpgdYR2mkKzR22HG9dMqb-1XbhDINyDNLmEA9g5p-5BI42CFl78mboUXkPM03OsMd2T3pCtkDGHdfl5HKuY2qhCQxVm7z7ksvn1XIonOAGz52ON-St88HuAU1S0NCk0w-xnKLY8.Te5hTwsXO9cCMz6Lan4KyVSWaGptj-XKh8Vaji8xcOE&dib_tag=se&keywords=sandisk+extreme+pro&qid=1708815146&sprefix=sandisk+extreme+pro%2Caps%2C163&sr=8-17

It’s the “V30” rating you need to look for.
This rating means the card is capable of a minimum of 30 MB/s write speeds, usually much higher. (They also have V60 and V90 rated cards that are even faster, but they are not cheap).
These cards are specifically designed for 4k and 8k ultra-high-def video recording on professional AV equipment. Thus they need to have very high throughput speeds.
Most SD cards and USB flash drives are advertised based on their read speeds. However, most of them have some horrendously slow write speeds (as I discovered after bench-marking all of my SD cards and USB flash drives)… but it is the write speeds that are much more important when using the card to run an operating system due to the constant high I/O throughput and random-access reads.

I actually bench-marked this V30 card on my desktop using Linux disk tools and a USB-3 card reader…
Read speeds were in excess of 300 MB/s.
More importantly, the write speeds were in excess of 80 MB/s.
Random access speeds were a mere 0.7 ms

All these specs WAY out-performed even a spinning hard drive connected with a USB-3 drive enclosure.

Since the Pi 4’s SD card reader is capable of the same speeds as a USB-3 port, it seems that this ultra-fast card is definitely making a very noticeable difference so far in how fast and smooth Mycodo now seems to be running on my Pi 4. I still haven’t enabled all of my Functions yet, but will be soon.
Ill post more updates as I continue testing.

[Lucid3y3 (Jason C LaFleur)]

UPDATE 2:

It’s been about a month now and I had all my hydroponics and hvac functions enabled and letting the Pi 4 run with the new faster SD card and I have not seen any of the problems I was having before with Mycodo losing measurement data and the database locking up. I did remove some unused Outputs and consolidated some of my Functions, as well as removing the Pi-camera all to help reduce the load on system resources as much as possible.

So just for comparison, I’ve switched back to the Pi 3B+ and just swapped the new fast SD card from the Pi 4 into the Pi 3B+. Initial performance seems to be a little slower as far as the WebGUI pages loading a bit slower than the Pi 4, But I want to see if using the faster card in the slower Pi fixes the problems I was having before. I’ll update this post again in about another month after testing this latest configuration.

[Lucid3y3 (Jason C LaFleur)]

Well I guess that answers that
I’m starting to wonder if the problem is Python itself… it’s just too slow and is a resource hog compared to something like C++.

File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/impl.py”, line 169, in _do_get
with util.safe_reraise():
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py”, line 146, in exit
raise exc_value.with_traceback(exc_tb)
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/impl.py”, line 167, in _do_get
return self._create_connection()
^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/base.py”, line 393, in _create_connection
return _ConnectionRecord(self)
^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/base.py”, line 678, in init
self.__connect()
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/base.py”, line 902, in __connect
with util.safe_reraise():
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py”, line 146, in exit
raise exc_value.with_traceback(exc_tb)
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/base.py”, line 898, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/engine/create.py”, line 645, in connect
return dialect.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/engine/default.py”, line 616, in connect
return self.loaded_dbapi.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sqlite3.OperationalError: unable to open database file

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/opt/Mycodo/mycodo/databases/utils.py”, line 26, in session_scope
yield session
File “/opt/Mycodo/mycodo/actions/base_action.py”, line 97, in setup_logger
Input.unique_id == action.function_id).first()
^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/orm/query.py”, line 2748, in first
return self.limit(1)._iter().first() # type: ignore
^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/orm/query.py”, line 2847, in _iter
result: Union[ScalarResult[_T], Result[_T]] = self.session.execute(
^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/orm/session.py”, line 2308, in execute
return self._execute_internal(
^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/orm/session.py”, line 2180, in _execute_internal
conn = self._connection_for_bind(bind)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/orm/session.py”, line 2047, in _connection_for_bind
return trans._connection_for_bind(engine, execution_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “”, line 2, in _connection_for_bind
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/orm/state_changes.py”, line 139, in _go
ret_value = fn(self, *arg, **kw)
^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/orm/session.py”, line 1143, in _connection_for_bind
conn = bind.connect()
^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/engine/base.py”, line 3269, in connect
return self._connection_cls(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/engine/base.py”, line 147, in init
Connection._handle_dbapi_exception_noconnection(
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/engine/base.py”, line 2431, in _handle_dbapi_exception_noconnection
raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/engine/base.py”, line 145, in init
self._dbapi_connection = engine.raw_connection()
^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/engine/base.py”, line 3293, in raw_connection
return self.pool.connect()
^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/base.py”, line 452, in connect
return _ConnectionFairy._checkout(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/base.py”, line 1269, in _checkout
fairy = _ConnectionRecord.checkout(pool)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/base.py”, line 716, in checkout
rec = pool._do_get()
^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/impl.py”, line 169, in _do_get
with util.safe_reraise():
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py”, line 146, in exit
raise exc_value.with_traceback(exc_tb)
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/impl.py”, line 167, in _do_get
return self._create_connection()
^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/base.py”, line 393, in _create_connection
return _ConnectionRecord(self)
^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/base.py”, line 678, in init
self.__connect()
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/base.py”, line 902, in __connect
with util.safe_reraise():
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py”, line 146, in exit
raise exc_value.with_traceback(exc_tb)
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/base.py”, line 898, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/engine/create.py”, line 645, in connect
return dialect.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/engine/default.py”, line 616, in connect
return self.loaded_dbapi.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file
(Background on this error at: Error Messages — SQLAlchemy 2.0 Documentation)
2024-03-16 08:01:09,526 - ERROR - mycodo.databases.utils - Error raised in session_scope. Session will be rolled back: db_uri=‘sqlite:////opt/Mycodo/databases/mycodo.db’, error=‘(sqlite3.OperationalError) unable to open database file
(Background on this error at: Error Messages — SQLAlchemy 2.0 Documentation)’
Traceback (most recent call last):
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/engine/base.py”, line 145, in init
self._dbapi_connection = engine.raw_connection()
^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/engine/base.py”, line 3293, in raw_connection
return self.pool.connect()
^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/base.py”, line 452, in connect
return _ConnectionFairy._checkout(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/base.py”, line 1269, in _checkout
fairy = _ConnectionRecord.checkout(pool)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/base.py”, line 716, in checkout
rec = pool._do_get()
^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/impl.py”, line 169, in _do_get
with util.safe_reraise():
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py”, line 146, in exit
raise exc_value.with_traceback(exc_tb)
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/impl.py”, line 167, in _do_get
return self._create_connection()
^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/base.py”, line 393, in _create_connection
return _ConnectionRecord(self)
^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/base.py”, line 678, in init
self.__connect()
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/base.py”, line 902, in __connect
with util.safe_reraise():
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py”, line 146, in exit
raise exc_value.with_traceback(exc_tb)
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/base.py”, line 898, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/engine/create.py”, line 645, in connect
return dialect.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/engine/default.py”, line 616, in connect
return self.loaded_dbapi.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sqlite3.OperationalError: unable to open database file

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/opt/Mycodo/mycodo/databases/utils.py”, line 26, in session_scope
yield session
File “/opt/Mycodo/mycodo/utils/database.py”, line 70, in db_retrieve_table_daemon
return_table = return_table.first()
^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/orm/query.py”, line 2748, in first
return self.limit(1)._iter().first() # type: ignore
^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/orm/query.py”, line 2847, in _iter
result: Union[ScalarResult[_T], Result[_T]] = self.session.execute(
^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/orm/session.py”, line 2308, in execute
return self._execute_internal(
^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/orm/session.py”, line 2180, in _execute_internal
conn = self._connection_for_bind(bind)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/orm/session.py”, line 2047, in _connection_for_bind
return trans._connection_for_bind(engine, execution_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “”, line 2, in _connection_for_bind
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/orm/state_changes.py”, line 139, in _go
ret_value = fn(self, *arg, **kw)
^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/orm/session.py”, line 1143, in _connection_for_bind
conn = bind.connect()
^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/engine/base.py”, line 3269, in connect
return self._connection_cls(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/engine/base.py”, line 147, in init
Connection._handle_dbapi_exception_noconnection(
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/engine/base.py”, line 2431, in _handle_dbapi_exception_noconnection
raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/engine/base.py”, line 145, in init
self._dbapi_connection = engine.raw_connection()
^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/engine/base.py”, line 3293, in raw_connection
return self.pool.connect()
^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/base.py”, line 452, in connect
return _ConnectionFairy._checkout(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/base.py”, line 1269, in _checkout
fairy = _ConnectionRecord.checkout(pool)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/base.py”, line 716, in checkout
rec = pool._do_get()
^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/impl.py”, line 169, in _do_get
with util.safe_reraise():
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py”, line 146, in exit
raise exc_value.with_traceback(exc_tb)
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/impl.py”, line 167, in _do_get
return self._create_connection()
^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/base.py”, line 393, in _create_connection
return _ConnectionRecord(self)
^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/base.py”, line 678, in init
self.__connect()
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/base.py”, line 902, in __connect
with util.safe_reraise():
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py”, line 146, in exit
raise exc_value.with_traceback(exc_tb)
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/pool/base.py”, line 898, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/engine/create.py”, line 645, in connect
return dialect.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/sqlalchemy/engine/default.py”, line 616, in connect
return self.loaded_dbapi.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file
(Background on this error at: Error Messages — SQLAlchemy 2.0 Documentation)
2024-03-16 08:01:09,751 - ERROR - mycodo.database - The Mycodo database is locked. Trying to access again in 1 second…
2024-03-16 08:01:09,585 - ERROR - mycodo.controllers.controller_conditional_8cb60bed - Exception executing Run Python Code
Traceback (most recent call last):
File “/opt/Mycodo/mycodo/controllers/controller_conditional.py”, line 183, in check_conditionals
self.conditional_run.conditional_code_run()
File “/opt/Mycodo/mycodo/user_python_code/conditional_8cb60bed-af46-471f-a1d5-faca410bee6f.py”, line 22, in conditional_code_run
measurement = self.condition(“a3855e9e”) #HS300 Lights Output Status LED Conditional
^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/mycodo/controllers/base_conditional.py”, line 78, in condition
return self.control.get_condition_measurement(full_cond_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/mycodo/mycodo_client.py”, line 190, in get_condition_measurement
return self.proxy().get_condition_measurement(condition_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/Pyro5/client.py”, line 510, in call
return self.__send(self.__name, args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/Pyro5/client.py”, line 256, in _pyroInvoke
msg = protocol.recv_stub(self._pyroConnection, [protocol.MSG_RESULT])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/Pyro5/protocol.py”, line 189, in recv_stub
header = connection.recv(6) # ‘PYRO’ + 2 bytes protocol version
^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/Pyro5/socketutil.py”, line 435, in recv
return receive_data(self.sock, size)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/Pyro5/socketutil.py”, line 159, in receive_data
raise err
Pyro5.errors.ConnectionClosedError: receiving: not enough data
2024-03-16 08:01:09,677 - ERROR - mycodo.database - The Mycodo database is locked. Trying to access again in 1 second…
2024-03-16 08:01:09,710 - ERROR - mycodo.controllers.controller_conditional_31d0988f - Exception executing Run Python Code
Traceback (most recent call last):
File “/opt/Mycodo/mycodo/controllers/controller_conditional.py”, line 183, in check_conditionals
self.conditional_run.conditional_code_run()
File “/opt/Mycodo/mycodo/user_python_code/conditional_31d0988f-3e3f-425e-89a9-2e14e830a3e3.py”, line 22, in conditional_code_run
measurement = self.condition(“f6538538”) #HS300 Pump Output Status LED Conditional
^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/mycodo/controllers/base_conditional.py”, line 78, in condition
return self.control.get_condition_measurement(full_cond_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/mycodo/mycodo_client.py”, line 190, in get_condition_measurement
return self.proxy().get_condition_measurement(condition_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/Pyro5/client.py”, line 510, in call
return self.__send(self.__name, args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/Pyro5/client.py”, line 268, in _pyroInvoke
data = serializer.loads(msg.data)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/Pyro5/serializers.py”, line 290, in loads
return self.recreate_classes(serpent.loads(data))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/Pyro5/serializers.py”, line 256, in recreate_classes
return self.dict_to_class(literal)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/Pyro5/serializers.py”, line 309, in dict_to_class
return super(SerpentSerializer, cls).dict_to_class(data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/Mycodo/env/lib/python3.11/site-packages/Pyro5/serializers.py”, line 235, in dict_to_class

So after deactivating all of my conditionals that run my LED status board it seems to be running ok. I just don’t understand how the simple conditionals can be causing such a huge CPU load.

VPD conditional…
Full Conditional code:

1: import os
2: import sys
3: sys.path.append(os.path.abspath(‘/var/mycodo-root’))
4: from mycodo.controllers.base_conditional import AbstractConditional
5: from mycodo.mycodo_client import DaemonControl
6: control = DaemonControl(pyro_timeout=30.0)
7:
8: from time import sleep
9: from datetime import datetime
10:
11: class ConditionalRun(AbstractConditional):
12: def init(self, logger, function_id, message):
13: super().init(logger, function_id, message, timeout=30.0)
14:
15: self.logger = logger
16: self.function_id = function_id
17: self.variables = {}
18: self.message = message
19: self.running = True
20: self.loop_count = 0
21:
22: def conditional_code_run(self):
23: blinks=8
24: measurement = self.condition(“1c8c49f3”) #VPD [Function24] Status LED Conditional
25: if measurement is not None: # If a measurement exists
26: if measurement < 0.8: # led is dark blue
27: self.run_action(“1a811689”, value={“payload”: “0,0,2”})
28: elif measurement >= 0.8 and measurement < 1.15: # led is green
29: self.run_action(“1a811689”, value={“payload”: “0,1,0”})
30: elif measurement >= 1.15 and measurement < 1.4: # led is yellow
31: self.run_action(“1a811689”, value={“payload”: “1,1,0”})
32: elif measurement >= 1.4: # led is bright red
33: self.run_action(“1a811689”, value={“payload”: “2,0,0”})
34: else: # If no measurement exists led is flashing bright blue
35: for _ in range(blinks):
36: self.run_action(“1a811689”, value={“payload”: “0,0,10”})
37: sleep(.75)
38: self.run_action(“1a811689”, value={“payload”: “0,0,0”})
39: sleep(.75)
40:
41: def function_status(self):
42: # Example code to provide a return status for other controllers and widgets.
43: status_dict = {
44: ‘string_status’: f"The controller has looped {self.loop_count} times at {datetime.now()}",
45: ‘loop_count’: self.loop_count,
46: ‘error’:
47: }
48: return status_dict

Humidity conditional…
Full Conditional code:

1: import os
2: import sys
3: sys.path.append(os.path.abspath(‘/var/mycodo-root’))
4: from mycodo.controllers.base_conditional import AbstractConditional
5: from mycodo.mycodo_client import DaemonControl
6: control = DaemonControl(pyro_timeout=30.0)
7:
8: from time import sleep
9: from datetime import datetime
10:
11: class ConditionalRun(AbstractConditional):
12: def init(self, logger, function_id, message):
13: super().init(logger, function_id, message, timeout=30.0)
14:
15: self.logger = logger
16: self.function_id = function_id
17: self.variables = {}
18: self.message = message
19: self.running = True
20: self.loop_count = 0
21:
22: def conditional_code_run(self):
23: blinks=8
24: measurement = self.condition(“5bdca006”) #SHTC3 (Grow Humidity Status LED Conditional)
25: if measurement is not None: # If a measurement exists
26: if measurement < 55: # If the measurement is < 55 led is dark blue
27: self.run_action(“9c798bf5”, value={“payload”: “0,0,2”})
28: elif measurement >= 55 and measurement < 68: # LED is green
29: self.run_action(“9c798bf5”, value={“payload”: “0,1,0”})
30: elif measurement >= 68 and measurement < 73: # LED is yellow
31: self.run_action(“9c798bf5”, value={“payload”: “1,1,0”})
32: elif measurement >= 73 and measurement < 75: # LED is red
33: self.run_action(“9c798bf5”, value={“payload”: “2,0,0”})
34: elif measurement >= 75: # Else If measurement is >= 75 LED is flashing bright red
35: for _ in range(blinks):
36: self.run_action(“9c798bf5”, value={“payload”: “5,0,0”})
37: sleep(.75)
38: self.run_action(“9c798bf5”, value={“payload”: “0,0,0”})
39: sleep(.75)
40: else: # If no measurement exists led is flashing bright blue
41: for _ in range(blinks):
42: self.run_action(“9c798bf5”, value={“payload”: “0,0,10”})
43: sleep(.75)
44: self.run_action(“9c798bf5”, value={“payload”: “0,0,0”})
45: sleep(.75)
46:
47: def function_status(self):
48: # Example code to provide a return status for other controllers and widgets.
49: status_dict = {
50: ‘string_status’: f"The controller has looped {self.loop_count} times at {datetime.now()}",
51: ‘loop_count’: self.loop_count,
52: ‘error’:
53: }
54: return status_dict

Temp conditional…
Full Conditional code:

1: import os
2: import sys
3: sys.path.append(os.path.abspath(‘/var/mycodo-root’))
4: from mycodo.controllers.base_conditional import AbstractConditional
5: from mycodo.mycodo_client import DaemonControl
6: control = DaemonControl(pyro_timeout=30.0)
7:
8: from time import sleep
9: from datetime import datetime
10:
11: class ConditionalRun(AbstractConditional):
12: def init(self, logger, function_id, message):
13: super().init(logger, function_id, message, timeout=30.0)
14:
15: self.logger = logger
16: self.function_id = function_id
17: self.variables = {}
18: self.message = message
19: self.running = True
20: self.loop_count = 0
21:
22: def conditional_code_run(self):
23: blinks=8
24: measurement = self.condition(“afe3b81a”) #SHTC3 (Grow Temp Status LED Conditional)
25: if measurement is not None: # If a measurement exists
26: if measurement < 55: # If the measurement is < 55 led is dark blue
27: self.run_action(“cba8014f”, value={“payload”: “0,0,2”})
28: elif measurement >= 55 and measurement < 60 : # Else If measurement is >= 55 led is medium blue
29: self.run_action(“cba8014f”, value={“payload”: “0,1,2”})
30: elif measurement >= 60 and measurement < 70: # Else If measurement is >= 60 led is light blue
31: self.run_action(“cba8014f”, value={“payload”: “0,2,1”})
32: elif measurement >= 70 and measurement < 80: # Else If measurement is >= 70 led is green
33: self.run_action(“cba8014f”, value={“payload”: “0,1,0”})
34: elif measurement >= 80 < 85: # Else If measurement is >= 80 led is yellow
35: self.run_action(“cba8014f”, value={“payload”: “1,1,0”})
36: elif measurement >= 85: # Else If measurement is >= 85 led is flashing bright red
37: for _ in range(blinks):
38: self.run_action(“cba8014f”, value={“payload”: “5,0,0”})
39: sleep(.75)
40: self.run_action(“cba8014f”, value={“payload”: “0,0,0”})
41: sleep(.75)
42: else: # If no measurement exists led is flashing bright blue
43: for _ in range(blinks):
44: self.run_action(“cba8014f”, value={“payload”: “0,0,10”})
45: sleep(.75)
46: self.run_action(“cba8014f”, value={“payload”: “0,0,0”})
47: sleep(.75)
48:
49: def function_status(self):
50: # Example code to provide a return status for other controllers and widgets.
51: status_dict = {
52: ‘string_status’: f"The controller has looped {self.loop_count} times at {datetime.now()}",
53: ‘loop_count’: self.loop_count,
54: ‘error’:
55: }
56: return status_dict

[Alpha]

Upgrading to V30 speed class SD cards changed exactly nothing for. In my opinion the card’s speed is irrelevant for the old Pi 3.
I found a solution for me. Pi 5 with 8GB RAM and SSD. Overkill, but works like a charm.

[Lucid3y3 (Jason C LaFleur)]

The faster card will only work if you remember to increase the clock speed of the Pi 3’s card reader by adding the following line to your config.txt file…
dtparam=sd_overclock=100

But upgrading to a better Pi works better

[Lucid3y3 (Jason C LaFleur)]

Finally got myself an M.2 SSD and moved Mycodo back to my Pi4 2gb.
The R/W specs on this SSD are about 2-3x faster than the V30 SD card I was previously using, verified with a benchmark test…

M.2 SSD…

V30 SD card…

The Pi boots a lot faster from the SSD, and the web interface is definitely more snappy than it’s ever been… still takes about 10-12 seconds to open my dashboards though
Now I’ll have to wait and see if eliminating the storage I/O bottleneck fixes all the problems I was having with my conditional controllers causing the system to bog down and crash.