Sudden malfunction "Error Pyroserver object has no attribute logger"

Mycodo
1

Hi, the problem started with web pages stopping refreshing properly, I have a tab for outputs, functions, logs and live measurements. Then function frames has started turning to brown has when it mycodo cannot update properly.

Then controllers stopped functionning properly, I had to shoot an emergency stop. I checked the system ressources which showed correct (memory, cpu, disk).

I have rebooted the Raspberry 4B, things have looked like coming back to normal functionning. Then When I start a controller (not modified), I had the error: “Error Pyroserver object has no attribute logger”. I don’t use pyro servers.

Mycodo Version: 8.15.8
Python Version: 3.7.3 (default, Oct 31 2022, 14:04:00) [GCC 8.3.0]
Linux raspberrypi 5.10.103-v7l+ #1529 SMP Tue Mar 8 12:24:00 GMT 2022 armv7l GNU/Linux

Disk Usage: df -h
/dev/root 29G 16G 13G 55% /
devtmpfs 1,8G 0 1,8G 0% /dev
tmpfs 1,9G 25M 1,9G 2% /dev/shm
tmpfs 1,9G 201M 1,7G 11% /run
tmpfs 5,0M 4,0K 5,0M 1% /run/lock
tmpfs 1,9G 0 1,9G 0% /sys/fs/cgroup
/dev/mmcblk0p1 253M 49M 204M 20% /boot
tmpfs 384M 4,0K 384M 1% /run/user/1000

Memory: free -h
total used free shared buff/cache available
Mem: 3,7Gi 655Mi 2,0Gi 317Mi 1,2Gi 2,7Gi
Swap: 511Mi 0B 511Mi

In the logs, I have: “RuntimeError: can’t start new thread” which looks bad. Here are some extracts (all here):

A lot of time outs like this one:

Traceback (most recent call last):
  File "/var/mycodo-root/mycodo/controllers/controller_conditional.py", line 183, in check_conditionals
    self.conditional_run.conditional_code_run()
  File "/home/pi/Mycodo/mycodo/user_python_code/conditional_c80725e6-c802-4d48-8a86-d2712ef1c28a.py", line 31, in conditional_code_run
    measurement = self.condition("48d53109-24ce-4f70-bbf3-978f4c0a780f") # cuve niveau bas
  File "/var/mycodo-root/mycodo/controllers/base_conditional.py", line 78, in condition
    return self.control.get_condition_measurement(full_cond_id)
  File "/var/mycodo-root/mycodo/mycodo_client.py", line 184, in get_condition_measurement
    return self.proxy().get_condition_measurement(condition_id)
  File "/var/mycodo-root/env/lib/python3.7/site-packages/Pyro5/client.py", line 510, in __call__
    return self.__send(self.__name, args, kwargs)
  File "/var/mycodo-root/env/lib/python3.7/site-packages/Pyro5/client.py", line 256, in _pyroInvoke
    msg = protocol.recv_stub(self._pyroConnection, [protocol.MSG_RESULT])
  File "/var/mycodo-root/env/lib/python3.7/site-packages/Pyro5/protocol.py", line 189, in recv_stub
    header = connection.recv(6)  # 'PYRO' + 2 bytes protocol version
  File "/var/mycodo-root/env/lib/python3.7/site-packages/Pyro5/socketutil.py", line 435, in recv
    return receive_data(self.sock, size)
  File "/var/mycodo-root/env/lib/python3.7/site-packages/Pyro5/socketutil.py", line 140, in receive_data
    raise TimeoutError("receiving: timeout")
Pyro5.errors.TimeoutError: receiving: timeout
2023-08-05 09:30:22,199 - INFO - mycodo.controllers.controller_conditional_c80725e6 - Relevage
2023-08-05 09:30:29,890 - ERROR - mycodo.controllers.controller_conditional_c80725e6 - Exception executing Run Python Code
Traceback (most recent call last):
  File "/var/mycodo-root/env/lib/python3.7/site-packages/Pyro5/socketutil.py", line 132, in receive_data
    chunk = sock.recv(size, socket.MSG_WAITALL)
socket.timeout: timed out

Unsupported serialized class:

Traceback (most recent call last):
  File "/var/mycodo-root/mycodo/controllers/controller_conditional.py", line 183, in check_conditionals
    self.conditional_run.conditional_code_run()
  File "/home/pi/Mycodo/mycodo/user_python_code/conditional_960d9665-a389-4a3c-8330-b8e0ca4c0ad8.py", line 33, in conditional_code_run
    measurement = self.condition("b5f6a76b-df40-48d4-a0da-e7939a571286")
  File "/var/mycodo-root/mycodo/controllers/base_conditional.py", line 78, in condition
    return self.control.get_condition_measurement(full_cond_id)
  File "/var/mycodo-root/mycodo/mycodo_client.py", line 184, in get_condition_measurement
    return self.proxy().get_condition_measurement(condition_id)
  File "/var/mycodo-root/env/lib/python3.7/site-packages/Pyro5/client.py", line 510, in __call__
    return self.__send(self.__name, args, kwargs)
  File "/var/mycodo-root/env/lib/python3.7/site-packages/Pyro5/client.py", line 268, in _pyroInvoke
    data = serializer.loads(msg.data)
  File "/var/mycodo-root/env/lib/python3.7/site-packages/Pyro5/serializers.py", line 290, in loads
    return self.recreate_classes(serpent.loads(data))
  File "/var/mycodo-root/env/lib/python3.7/site-packages/Pyro5/serializers.py", line 256, in recreate_classes
    return self.dict_to_class(literal)
  File "/var/mycodo-root/env/lib/python3.7/site-packages/Pyro5/serializers.py", line 309, in dict_to_class
    return super(SerpentSerializer, cls).dict_to_class(data)
  File "/var/mycodo-root/env/lib/python3.7/site-packages/Pyro5/serializers.py", line 235, in dict_to_class
    raise errors.SerializeError("unsupported serialized class: " + classname)
Pyro5.errors.SerializeError: unsupported serialized class: urllib3.exceptions.NewConnectionError
2023-08-05 09:33:56,221 - INFO - mycodo.controllers.controller_conditional_960d9665 - Test Cuve Niv Haut
2023-08-05 09:34:06,089 - ERROR - mycodo.controllers.controller_conditional_960d9665 - Exception executing Run Python Code

Can’t start a new thread:

Traceback (most recent call last):
  File "/var/mycodo-root/mycodo/controllers/base_controller.py", line 81, in run
    self.loop()
  File "/var/mycodo-root/mycodo/controllers/controller_input.py", line 249, in loop
    use_same_timestamp=use_same_timestamp)
  File "/var/mycodo-root/mycodo/utils/influx.py", line 177, in add_measurements_influxdb
    write_db.start()
  File "/usr/lib/python3.7/threading.py", line 847, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
2

Whatever is going on, it’s pretty serious for your system to not be able to start a new thread. I don’t see the exception you mention in the title pasted, which is arguably the most important.

3

Example lines 946, 957, 968, 979 in the pastbin

4

Can you paste it in the thread? The internet is filled with forums linked to dead 3rd party sites and links, and it will eventually happen here if you give it enough time.

5

Sure, here it is. Tell me if it is enough:


2023-08-05 09:37:22,298 - ERROR - mycodo.controllers.controller_input_33161914 - loop() Error
Traceback (most recent call last):
  File "/var/mycodo-root/mycodo/controllers/base_controller.py", line 81, in run
    self.loop()
  File "/var/mycodo-root/mycodo/controllers/controller_input.py", line 249, in loop
    use_same_timestamp=use_same_timestamp)
  File "/var/mycodo-root/mycodo/utils/influx.py", line 177, in add_measurements_influxdb
    write_db.start()
  File "/usr/lib/python3.7/threading.py", line 847, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
2023-08-05 09:37:23,120 - ERROR - mycodo.controllers.controller_input_32248638 - loop() Error
Traceback (most recent call last):
  File "/var/mycodo-root/mycodo/controllers/base_controller.py", line 81, in run
    self.loop()
  File "/var/mycodo-root/mycodo/controllers/controller_input.py", line 249, in loop
    use_same_timestamp=use_same_timestamp)
  File "/var/mycodo-root/mycodo/utils/influx.py", line 177, in add_measurements_influxdb
    write_db.start()
  File "/usr/lib/python3.7/threading.py", line 847, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
2023-08-05 09:37:23,138 - ERROR - mycodo.controllers.controller_input_fd75df2f - loop() Error
Traceback (most recent call last):
  File "/var/mycodo-root/mycodo/controllers/base_controller.py", line 81, in run
    self.loop()
  File "/var/mycodo-root/mycodo/controllers/controller_input.py", line 249, in loop
    use_same_timestamp=use_same_timestamp)
  File "/var/mycodo-root/mycodo/utils/influx.py", line 177, in add_measurements_influxdb
    write_db.start()
  File "/usr/lib/python3.7/threading.py", line 847, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
6

In fact, since I have upgraded to Mycodo Version: 8.15.8, I could not make working any conditional controllers - every thing else looks fine. I am trying to rebuild one from scratch.

7

I still don’t see the error mentioned in the title.

8

I don’t find it myself in the log and have searched far to yesterday. So I am not sure, possibly only a notification.

9

Can you replicate the behavior, and if so, can you provide a concise list of steps?

10

I cannot, there are too much problems : live measurements don’t update, measurements at all are not updated since my conditional controllers don’t detect conditions, etc. This looks like either my system or mycodo is broken. But if I restart the backend, it works again for a while.