Error trying to fetch: Python int too large to convert to C long

Started by franklinoffner, October 25, 2020, 04:08:57 PM

Previous topic - Next topic

grumpy_geoff

Quote from: prl on October 26, 2020, 10:18:39 PM
Strange. Tonight I'm only seeing the expected errors on "Politics" (ACT guide).

Plugin version is 20191013 (as the T4 is on the beta).  I guess that's why you're not seeing them.

2,511 entries were reported in this morning's boot from deep standby -
{1000}<   141.523> [IceTV] 2020-10-27 07:12:06: Start update
{1000}<   147.121> [EPGFetcher] ERROR: lookup of 0x10 "Drama" returned "Movie"
{1000}<   147.124> [EPGFetcher] ERROR: lookup of 0xa0 "Lifestyle" returned "Infotainment"
...
{1000}<   159.571> [EPGFetcher] ERROR: lookup of 0x11 "Mystery" returned "Crime"
{1000}<   159.573> [EPGFetcher] ERROR: lookup of 0x10 "Drama" returned "Movie"
{1000}<   160.287> [IceTV] 2020-10-27 07:12:25: EPG download OK
{1000}<   160.294> [IceTV] 2020-10-27 07:12:25: Timers updated OK
{1000}<   160.294> [IceTV] 2020-10-27 07:12:25: End update



Last night, when the Wiz fetched the guide data for the newly published day (Sun), there were 269 "lookup errors" reported.
The GUI had been running for 5.5 hours by then.

Us "far westies" must be getting special treatment ;0


prl

Quote from: grumpy_geoff on October 27, 2020, 10:42:30 AM
Plugin version is 20191013 (as the T4 is on the beta).  I guess that's why you're not seeing them.

I run the beta on everything, including our in-use box.
Peter
Beyonwiz T4 in-use
Beyonwiz T2, T3, T4, U4 & V2 for testing

prl

Quote from: grumpy_geoff on October 26, 2020, 10:35:47 PM
Strictly speaking it was the second fetch, as the first failed with the known "lock held" issue -
  File "/usr/lib/enigma2/python/Plugins/SystemPlugins/IceTV/plugin.py", line 514, in doWork
  File "/usr/lib/enigma2/python/Plugins/Extensions/IceTVDesc/hook.py", line 14, in run
  File "/usr/lib/enigma2/python/Plugins/Extensions/IceTVDesc/hook.py", line 31, in makeChanShowMap
exceptions.ImportError: Failed to import _strptime because the import lockis held by another thread.


You need to talk to adoxa about that: it's in his plugin.
Peter
Beyonwiz T4 in-use
Beyonwiz T2, T3, T4, U4 & V2 for testing

prl

Quote from: grumpy_geoff on October 26, 2020, 06:49:10 PM
Quote from: prl on October 26, 2020, 05:11:51 PM
The genre errors that grumpy_geoff reported, though, seem to be due to whatever was happening to the IceTV server yesterday, possibly down to strange things in the EPG, but unfortunately there wasn't enough time to dig into the problem to nail down the exact cause (and I never saw those errors).
{12595}< 16282.929> [EPGFetcher] ERROR: lookup of 0x10 "Drama" returned "Movie"
{12595}< 16282.950> [EPGFetcher] ERROR: lookup of 0x10 "Drama" returned "Movie"
{12595}< 16282.954> [EPGFetcher] ERROR: lookup of 0x10 "Drama" returned "Movie"
...

I suspect that adoxa's IceTVDesc (or at least the version that you're using) hasn't been updated to take account of the fact that IceTV now has multilingual genre support. I can't find the source code on either his Birbucket or Github accounts to check.

Try uninstalling it and see whether the errors (apart from the expected one on "Politics") go away.
Peter
Beyonwiz T4 in-use
Beyonwiz T2, T3, T4, U4 & V2 for testing

grumpy_geoff

Quote from: prl on October 27, 2020, 11:01:55 AM
Quote from: grumpy_geoff on October 26, 2020, 10:35:47 PM
Strictly speaking it was the second fetch, as the first failed with the known "lock held" issue -
  File "/usr/lib/enigma2/python/Plugins/SystemPlugins/IceTV/plugin.py", line 514, in doWork
  File "/usr/lib/enigma2/python/Plugins/Extensions/IceTVDesc/hook.py", line 14, in run
  File "/usr/lib/enigma2/python/Plugins/Extensions/IceTVDesc/hook.py", line 31, in makeChanShowMap
exceptions.ImportError: Failed to import _strptime because the import lockis held by another thread.


You need to talk to adoxa about that: it's in his plugin.

It's the same locking conflict with the import of the _strptime module whether IceTVDesc is used or not.

grumpy_geoff

Quote from: prl on October 27, 2020, 11:13:40 AM
I suspect that adoxa's IceTVDesc (or at least the version that you're using) hasn't been updated to take account of the fact that IceTV now has multilingual genre support. I can't find the source code on either his Birbucket or Github accounts to check.

Try uninstalling it and see whether the errors (apart from the expected one on "Politics") go away.

Yep, that is it.  I only had the one lookup failure -
{5954}<  5117.459> [EPGFetcher] ERROR: lookup of 0x01 "Politics" returned "Cult"


prl

Quote from: grumpy_geoff on October 27, 2020, 11:47:16 AM
Quote from: prl on October 27, 2020, 11:01:55 AM
Quote from: grumpy_geoff on October 26, 2020, 10:35:47 PM
Strictly speaking it was the second fetch, as the first failed with the known "lock held" issue -
  File "/usr/lib/enigma2/python/Plugins/SystemPlugins/IceTV/plugin.py", line 514, in doWork
  File "/usr/lib/enigma2/python/Plugins/Extensions/IceTVDesc/hook.py", line 14, in run
  File "/usr/lib/enigma2/python/Plugins/Extensions/IceTVDesc/hook.py", line 31, in makeChanShowMap
exceptions.ImportError: Failed to import _strptime because the import lockis held by another thread.


You need to talk to adoxa about that: it's in his plugin.

It's the same locking conflict with the import of the _strptime module whether IceTVDesc is used or not.

Strange. I don't see any instances of it over a week's worth of logs on our in-use T4.
Peter
Beyonwiz T4 in-use
Beyonwiz T2, T3, T4, U4 & V2 for testing


prl

grumpy_geoff has confirmed (on the Beyonwiz forum) that his genre errors other than for "Politics" are due to the IceTVDesc plugin (which adds extra metadata, like credits, to the displayed extended description).

I've pointed adoxa (the plugin's author) to this discussion.
Peter
Beyonwiz T4 in-use
Beyonwiz T2, T3, T4, U4 & V2 for testing

prl

A new version of the plugin with the updated IceTV Genre tables has been merged into the source repository and should appear in the next release. Look for plugin version (in the MENU>IceTV heading) 20201026.

In the meantime, IceTV has started using a second new genre that's not handled by the current plugin (version 20191013), "History". Until the new plugin is available, expect to see logging like:
{3718}<  1839.374> [EPGFetcher] ERROR: lookup of 0x01 "Politics" returned "Cult"
{3718}<  1840.008> [EPGFetcher] ERROR: lookup of 0x01 "History" returned "Cult"
in the debug logs.

Note that when error messages like that indicate that there is a genre mapping failure in the plugin, the problematic genre in not added to the EPG (and not included in any recordings made of the affected program), not that the wrong genre is added to the EPG (and to affected recordings).
Peter
Beyonwiz T4 in-use
Beyonwiz T2, T3, T4, U4 & V2 for testing

peteru

There are still some IceTV users that report problems with their Beyonwiz machines.

Has anyone at IceTV figured out what went wrong? What was the nature of the data being sent that caused the denial of service condition on the Beyonwiz machines? I'd like to put in a proper fix so that this does not keep on re-occurring.

I don't like the idea of shrugging the shoulders and saying it was fixed by turning it off and on again. It would be good to know if this is an inherent protocol/API issue, data validation issue, algorithm selection issue or just a plain old bug that needs specific conditions to trigger.

IanL-S

Peter, I cannot be of assistance. I am not aware of this happening with any of my T2s or my U4. I have not looked at the logs. If it will help I will do so.

Ian
IceTV: IceBox + BYOB IceBox + 2xTRF-2400 + 2xTF7100HDPVRtPlus + SKIPPA [RIP] + T2 + U4 + V2
No IceTV: a few Toppys and T2
Synology NAS
Check out the oztoppy wiki and oztoppy Forum for Toppy help

MD

Peter,

We do not know of anyone who still has the issue. If you do maybe you could illuminate us as to who so we can follow up? At the time of the alert Daniel and prl spent about three hours tearing out their respective hair but could not find any reason for the problem.  In the end an API restart was tried and solved it but in doing that we lost any data to continue our examination.  We did not shrug our shoulders and ignore it but commercially we can't tie up resources trying to find an obscure bug that has never appeared before and may not again. If it re-appears we will be a little more forewarned and will take appropriate action at the time.

peteru

Quote from: MD on October 30, 2020, 12:31:03 PMWe do not know of anyone who still has the issue. If you do maybe you could illuminate us as to who so we can follow up?

IanL-S reported instability yesterday and was looking to configure his systems to regularly reboot as a workaround for issues.

One of the problems here is that there is no obvious correlation between the symptoms that users experience and IceTV. The vast majority of users will not look at logs or suspect the IceTV plugin. The symptoms vary. Because the machine runs out of resources, failures can occur in any part of the code, resulting in unpredictable problems. Often these problems do not generate any logs. This lead a number of users to suspect hardware issues.

Quote from: MD on October 30, 2020, 12:31:03 PMWe did not shrug our shoulders and ignore it

I was not suggesting you did. I mean that I don't want to dismiss this as a non-issue for the Beyonwiz plugin code. I'd like to find the root cause and make the Beyonwiz plugin code more resilient. However, that is not possible without any clues as to what may have gone wrong. Some samples of the data that caused the issues would probably go a long way towards narrowing down the problem. Even server log information may provide some clues - were the responses larger than usual?

prl

Hi, Colin. I haven't heard of any more reports of the "Error trying to fetch: Python int too large to convert to C long" problem since Daniel restarted the server at about 17:00 on Sun 25 Oct.

On Sunday, Daniel restarted the IceTV server while I was trying to find the cause of the "int too large" problem. I understand that he had a pressing need to get things working again, but it meant that I didn't get any useful debug information about the problem.

There have been a number of reports of problems with IceTV causing UI lockups on Beyonwizes, but there is only one that has any level of detail, from csutak40 (Judy). It shows evidence of possible out-of-memory issues, but none of them during the times on 24/5 Oct when the IceTV server was known to be causing the "int too large" problems. Some of csutak40's recent crashes were due to her T2's HDD filling, and those don't appear to be associated with IceTV in any way.

Other users report the UI (often of T2s) seizing up for no apparent reason, but so far,  csutak40 is the only person to have provided any potentially useful debug information, but unfortunately there was little in that that is much help in localising the problem.

Ianl-S's post about his recent problems has really nothing in it that clearly either implicates or absolves the IceTV plugin, or its interaction with the server.

Grumpy_Geoff and I have both tried to replicate the problem, he on his in-use T2 and I on my test T2, but without any success.
Peter
Beyonwiz T4 in-use
Beyonwiz T2, T3, T4, U4 & V2 for testing