Not getting updates and getting this entry in the ice log after attempting a manual fetch and update.
This is on a Beyonwiz U4 which has been rebooted.
Tried enable/disable to no avail.
Any ideas what to try next?
IceTV version 20191013
Alan
This is definitely a weird one, and we have had other reports of this as well and are currently investigating.
OK, had a look into the Wiz log file and find this
{868}< 39.497> Unhandled Error
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
File "/usr/lib/python2.7/threading.py", line 754, in run
File "/usr/lib/python2.7/site-packages/twisted/_threads/_threadworker.py", line 46, in work
File "/usr/lib/python2.7/site-packages/twisted/_threads/_team.py", line 190, in doWork
--- <exception caught here> ---
File "/usr/lib/python2.7/site-packages/twisted/python/threadpool.py", line 250, in inContext
File "/usr/lib/python2.7/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
File "/usr/lib/python2.7/site-packages/twisted/python/context.py", line 122, in callWithContext
File "/usr/lib/python2.7/site-packages/twisted/python/context.py", line 85, in callWithContext
File "/usr/lib/enigma2/python/Plugins/SystemPlugins/IceTV/plugin.py", line 517, in doWork
exceptions.OverflowError: Python int too large to convert to C long
Trust that is might help
Alan
Incase that it is relevant the U4 is running current beta
System OE: OE-Alliance 4.3
Firmware version: beyonwiz 19.3.20201011 (19.3-346-g6461e3a)
Kernel / Drivers: 4.9.51 / 180411-16.4
I believe the issue started occurring last evening (AWST).
Did IceTV add a new genre?
{780}< 46871.718> [IceTV] 2020-10-24 20:56:11: EPG download OK
{780}< 46871.718> [IceTV] 2020-10-24 20:56:11: Timers updated OK
{780}< 46871.719> [IceTV] 2020-10-24 20:56:11: End update
...
{780}< 47770.899> [IceTV] 2020-10-24 21:11:11: Start update
{780}< 47771.889> [EPGFetcher] ERROR: lookup of 0x20 "Current Affairs" returned "News"
{780}< 47771.897> [EPGFetcher] ERROR: lookup of 0x10 "Drama" returned "Movie"
{780}< 47771.900> [EPGFetcher] ERROR: lookup of 0x10 "Drama" returned "Movie"
{780}< 47771.901> [EPGFetcher] ERROR: lookup of 0x11 "Mystery" returned "Crime"
{780}< 47771.901> [EPGFetcher] ERROR: lookup of 0x01 "Violence" returned "Cult"
...
{780}< 47773.568> [EPGFetcher] ERROR: lookup of 0x01 "Renovation" returned "Cult"
{780}< 47773.573> [EPGFetcher] ERROR: lookup of 0xa0 "Lifestyle" returned "Infotainment"
{780}< 47773.576> [EPGFetcher] ERROR: lookup of 0xa0 "Lifestyle" returned "Infotainment"
{780}< 47773.578> [EPGFetcher] ERROR: lookup of 0xa0 "Lifestyle" returned "Infotainment"
{780}< 47773.579> [EPGFetcher] ERROR: lookup of 0x01 "Renovation" returned "Cult"
{780}< 47773.629> /usr/lib/enigma2/python/Plugins/SystemPlugins/IceTV/plugin.py:517: RuntimeWarning: tp_compare didn't return -1 or -2 for exception
{780}< 47773.649> Unhandled Error
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
File "/usr/lib/python2.7/threading.py", line 754, in run
File "/usr/lib/python2.7/site-packages/twisted/_threads/_threadworker.py", line 46, in work
File "/usr/lib/python2.7/site-packages/twisted/_threads/_team.py", line 190, in doWork
--- <exception caught here> ---
File "/usr/lib/python2.7/site-packages/twisted/python/threadpool.py", line 250, in inContext
File "/usr/lib/python2.7/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
File "/usr/lib/python2.7/site-packages/twisted/python/context.py", line 122, in callWithContext
File "/usr/lib/python2.7/site-packages/twisted/python/context.py", line 85, in callWithContext
File "/usr/lib/enigma2/python/Plugins/SystemPlugins/IceTV/plugin.py", line 517, in doWork
exceptions.OverflowError: Python int too large to convert to C long
We have not added a new genre, but it is also now fixed, we will keep investigating to try and find the root cause, but it is now working.
I can confirm working. EPG and timers now loaded on our U4.
Many thanks for such quick response.
Alan.
+1. :)
both my devices are now updating ok now.
thanks.
Phew.
I think the issue is back again. First sign was frozen remote, then spinning wheel and then reboot.
The T2 has been running continuously since I re-enabled ICETV around 3pm. If it is an issue to do with memory creep/depletion, this maybe of some help in nailing down the problem.
Quote from: JPP on October 25, 2020, 09:10:07 PM
I think the issue is back again. First sign was frozen remote, then spinning wheel and then reboot.
The T2 has been running continuously since I re-enabled ICETV around 3pm. If it is an issue to do with memory creep/depletion, this maybe of some help in nailing down the problem.
I tested late last night and could not replicate it (which I could earlier in the day), is there anything in the debug logs.
Both my U4 and T3 were left in standby all night, each with multiple recordings scheduled during the night.
All indications this morning were that there were no disruptions in Ice log, and all recordings competed with no sign of disruption.
BTW, I have my Ice interval set at 15 minutes. Probably too much information, but just in case someone might have mentally gone down that path
Since around 8AM today I have put them both to normal use. Just rechecked (at 10 AM) once again with no disruptions
Fingers crossed.
IceTV have added quite a few new genres, but currently only one of the new genres is being used in the EPG ("Politics", currently being used on "QLD Votes 2020" and "Biden V Trump: The Choice 2020").
Somehow, the "Motorsport" genre also got added, when there was already a "Motor Sport" genre. The "Motorsport" genre has been removed.
The genre errors that grumpy_geoff reported, though, seem to be due to whatever was happening to the IceTV server yesterday, possibly down to strange things in the EPG, but unfortunately there wasn't enough time to dig into the problem to nail down the exact cause (and I never saw those errors).
The debug log errors that the "Motorsport" and "Politics" genres caused were:
{642}< 7279.226> [EPGFetcher] ERROR: lookup of 0x47 "Motorsport" returned "Motor Sport"
{642}< 7285.871> [EPGFetcher] ERROR: lookup of 0x01 "Politics" returned "Cult"
Now that the "Motorsport" genre has been removed, I'm now only getting the error for the "Politics" genre. The error is handled by ignoring the unrecognised genre and not putting it into the EPG.
I have submitted an update of the IceTV plugin with the updated genre tables to the Beyonwiz code repository.
Quote from: prl on October 26, 2020, 05:11:51 PM
The genre errors that grumpy_geoff reported, though, seem to be due to whatever was happening to the IceTV server yesterday, possibly down to strange things in the EPG, but unfortunately there wasn't enough time to dig into the problem to nail down the exact cause (and I never saw those errors).
I still get them, e.g. just now there were 2,612 "[EPGFetcher]" entries -
{12595}< 16279.068> [IceTV] 2020-10-26 15:41:05: Start update
{12595}< 16282.929> [EPGFetcher] ERROR: lookup of 0x10 "Drama" returned "Movie"
{12595}< 16282.950> [EPGFetcher] ERROR: lookup of 0x10 "Drama" returned "Movie"
{12595}< 16282.954> [EPGFetcher] ERROR: lookup of 0x10 "Drama" returned "Movie"
...
{12595}< 16295.572> [EPGFetcher] ERROR: lookup of 0x01 "Cycling" returned "Cult"
{12595}< 16295.581> [EPGFetcher] ERROR: lookup of 0x20 "Current Affairs" returned "News"
{12595}< 16295.582> [EPGFetcher] ERROR: lookup of 0x01 "Politics" returned "Cult"
{12595}< 16296.289> [IceTV] 2020-10-26 15:41:23: EPG download OK
{12595}< 16296.322> [RecordTimer] ignore double timer
{12595}< 16296.323> [IceTV] 2020-10-26 15:41:23: Timers updated OK
{12595}< 16296.324> [IceTV] 2020-10-26 15:41:23: End update
Here's the unique set:
[EPGFetcher] ERROR: lookup of 0x01 "Cycling" returned "Cult"
[EPGFetcher] ERROR: lookup of 0x01 "Family" returned "Cult"
[EPGFetcher] ERROR: lookup of 0x01 "Horse Racing" returned "Cult"
[EPGFetcher] ERROR: lookup of 0x01 "Politics" returned "Cult"
[EPGFetcher] ERROR: lookup of 0x01 "Renovation" returned "Cult"
[EPGFetcher] ERROR: lookup of 0x01 "Short Film" returned "Cult"
[EPGFetcher] ERROR: lookup of 0x01 "Special" returned "Cult"
[EPGFetcher] ERROR: lookup of 0x01 "Violence" returned "Cult"
[EPGFetcher] ERROR: lookup of 0x01 "Youth" returned "Cult"
[EPGFetcher] ERROR: lookup of 0x10 "Drama" returned "Movie"
[EPGFetcher] ERROR: lookup of 0x11 "Murder" returned "Crime"
[EPGFetcher] ERROR: lookup of 0x11 "Mystery" returned "Crime"
[EPGFetcher] ERROR: lookup of 0x11 "Thriller" returned "Crime"
[EPGFetcher] ERROR: lookup of 0x12 "Action" returned "Adventure"
[EPGFetcher] ERROR: lookup of 0x12 "War" returned "Adventure"
[EPGFetcher] ERROR: lookup of 0x12 "Western" returned "Adventure"
[EPGFetcher] ERROR: lookup of 0x13 "Fantasy" returned "Sci-Fi"
[EPGFetcher] ERROR: lookup of 0x13 "Horror" returned "Sci-Fi"
[EPGFetcher] ERROR: lookup of 0x14 "Sitcom" returned "Comedy"
[EPGFetcher] ERROR: lookup of 0x20 "Current Affairs" returned "News"
[EPGFetcher] ERROR: lookup of 0x45 "AFL" returned "Football"
[EPGFetcher] ERROR: lookup of 0x45 "American Football" returned "Football"
[EPGFetcher] ERROR: lookup of 0x45 "Baseball" returned "Football"
[EPGFetcher] ERROR: lookup of 0x45 "Cricket" returned "Football"
[EPGFetcher] ERROR: lookup of 0x45 "Netball" returned "Football"
[EPGFetcher] ERROR: lookup of 0x45 "Rugby League" returned "Football"
[EPGFetcher] ERROR: lookup of 0x45 "Rugby" returned "Football"
[EPGFetcher] ERROR: lookup of 0x55 "Cartoon" returned "Animation"
[EPGFetcher] ERROR: lookup of 0x60 "Dance" returned "Music"
[EPGFetcher] ERROR: lookup of 0x60 "Musical" returned "Music"
[EPGFetcher] ERROR: lookup of 0x80 "Business & Finance" returned "Society & Culture"
[EPGFetcher] ERROR: lookup of 0x80 "Parliament" returned "Society & Culture"
[EPGFetcher] ERROR: lookup of 0x90 "Science" returned "Education"
[EPGFetcher] ERROR: lookup of 0xa0 "Fishing" returned "Infotainment"
[EPGFetcher] ERROR: lookup of 0xa0 "Lifestyle" returned "Infotainment"
[EPGFetcher] ERROR: lookup of 0xa5 "Food/Wine" returned "Cooking"
Strange. Tonight I'm only seeing the expected errors on "Politics" (ACT guide).
I assume that you've done a GUI restart (at least) since the problems on Sunday?
Quote from: prl on October 26, 2020, 10:18:39 PM
Strange. Tonight I'm only seeing the expected errors on "Politics" (ACT guide).
I assume that you've done a GUI restart (at least) since the problems on Sunday?
Yes, that was from the first fetch after a GUI restart this 'arvo.
Strictly speaking it was the second fetch, as the first failed with the known "lock held" issue -
File "/usr/lib/enigma2/python/Plugins/SystemPlugins/IceTV/plugin.py", line 514, in doWork
File "/usr/lib/enigma2/python/Plugins/Extensions/IceTVDesc/hook.py", line 14, in run
File "/usr/lib/enigma2/python/Plugins/Extensions/IceTVDesc/hook.py", line 31, in makeChanShowMap
exceptions.ImportError: Failed to import _strptime because the import lockis held by another thread.
Quote from: prl on October 26, 2020, 10:18:39 PM
Strange. Tonight I'm only seeing the expected errors on "Politics" (ACT guide).
Plugin version is 20191013 (as the T4 is on the beta). I guess that's why you're not seeing them.
2,511 entries were reported in this morning's boot from deep standby -
{1000}< 141.523> [IceTV] 2020-10-27 07:12:06: Start update
{1000}< 147.121> [EPGFetcher] ERROR: lookup of 0x10 "Drama" returned "Movie"
{1000}< 147.124> [EPGFetcher] ERROR: lookup of 0xa0 "Lifestyle" returned "Infotainment"
...
{1000}< 159.571> [EPGFetcher] ERROR: lookup of 0x11 "Mystery" returned "Crime"
{1000}< 159.573> [EPGFetcher] ERROR: lookup of 0x10 "Drama" returned "Movie"
{1000}< 160.287> [IceTV] 2020-10-27 07:12:25: EPG download OK
{1000}< 160.294> [IceTV] 2020-10-27 07:12:25: Timers updated OK
{1000}< 160.294> [IceTV] 2020-10-27 07:12:25: End updateLast night, when the Wiz fetched the guide data for the newly published day (Sun), there were 269 "lookup errors" reported.
The GUI had been running for 5.5 hours by then.
Us "far westies" must be getting special treatment ;0
Quote from: grumpy_geoff on October 27, 2020, 10:42:30 AM
Plugin version is 20191013 (as the T4 is on the beta). I guess that's why you're not seeing them.
I run the beta on everything, including our in-use box.
Quote from: grumpy_geoff on October 26, 2020, 10:35:47 PM
Strictly speaking it was the second fetch, as the first failed with the known "lock held" issue -
File "/usr/lib/enigma2/python/Plugins/SystemPlugins/IceTV/plugin.py", line 514, in doWork
File "/usr/lib/enigma2/python/Plugins/Extensions/IceTVDesc/hook.py", line 14, in run
File "/usr/lib/enigma2/python/Plugins/Extensions/IceTVDesc/hook.py", line 31, in makeChanShowMap
exceptions.ImportError: Failed to import _strptime because the import lockis held by another thread.
You need to talk to adoxa about that: it's in his plugin.
Quote from: grumpy_geoff on October 26, 2020, 06:49:10 PM
Quote from: prl on October 26, 2020, 05:11:51 PM
The genre errors that grumpy_geoff reported, though, seem to be due to whatever was happening to the IceTV server yesterday, possibly down to strange things in the EPG, but unfortunately there wasn't enough time to dig into the problem to nail down the exact cause (and I never saw those errors).
{12595}< 16282.929> [EPGFetcher] ERROR: lookup of 0x10 "Drama" returned "Movie"
{12595}< 16282.950> [EPGFetcher] ERROR: lookup of 0x10 "Drama" returned "Movie"
{12595}< 16282.954> [EPGFetcher] ERROR: lookup of 0x10 "Drama" returned "Movie"
...
I suspect that adoxa's IceTVDesc (or at least the version that you're using) hasn't been updated to take account of the fact that IceTV now has multilingual genre support. I can't find the source code on either his Birbucket or Github accounts to check.
Try uninstalling it and see whether the errors (apart from the expected one on "Politics") go away.
Quote from: prl on October 27, 2020, 11:01:55 AM
Quote from: grumpy_geoff on October 26, 2020, 10:35:47 PM
Strictly speaking it was the second fetch, as the first failed with the known "lock held" issue -
File "/usr/lib/enigma2/python/Plugins/SystemPlugins/IceTV/plugin.py", line 514, in doWork
File "/usr/lib/enigma2/python/Plugins/Extensions/IceTVDesc/hook.py", line 14, in run
File "/usr/lib/enigma2/python/Plugins/Extensions/IceTVDesc/hook.py", line 31, in makeChanShowMap
exceptions.ImportError: Failed to import _strptime because the import lockis held by another thread.
You need to talk to adoxa about that: it's in his plugin.
It's the same locking conflict with the import of the _strptime module whether IceTVDesc is used or not.
Quote from: prl on October 27, 2020, 11:13:40 AM
I suspect that adoxa's IceTVDesc (or at least the version that you're using) hasn't been updated to take account of the fact that IceTV now has multilingual genre support. I can't find the source code on either his Birbucket or Github accounts to check.
Try uninstalling it and see whether the errors (apart from the expected one on "Politics") go away.
Yep, that is it. I only had the one lookup failure -
{5954}< 5117.459> [EPGFetcher] ERROR: lookup of 0x01 "Politics" returned "Cult"
Quote from: grumpy_geoff on October 27, 2020, 11:47:16 AM
Quote from: prl on October 27, 2020, 11:01:55 AM
Quote from: grumpy_geoff on October 26, 2020, 10:35:47 PM
Strictly speaking it was the second fetch, as the first failed with the known "lock held" issue -
File "/usr/lib/enigma2/python/Plugins/SystemPlugins/IceTV/plugin.py", line 514, in doWork
File "/usr/lib/enigma2/python/Plugins/Extensions/IceTVDesc/hook.py", line 14, in run
File "/usr/lib/enigma2/python/Plugins/Extensions/IceTVDesc/hook.py", line 31, in makeChanShowMap
exceptions.ImportError: Failed to import _strptime because the import lockis held by another thread.
You need to talk to adoxa about that: it's in his plugin.
It's the same locking conflict with the import of the _strptime module whether IceTVDesc is used or not.
Strange. I don't see any instances of it over a week's worth of logs on our in-use T4.
Quote from: prl on October 27, 2020, 12:20:11 PM
Strange. I don't see any instances of it over a week's worth of logs on our in-use T4.
Yeah, it's very infrequent - https://www.beyonwiz.com.au/forum/viewtopic.php?f=55&t=13224#p176639
grumpy_geoff has confirmed (https://beyonwiz.com.au/forum/viewtopic.php?f=16&p=190702#p190702) (on the Beyonwiz forum) that his genre errors other than for "Politics" are due to the IceTVDesc plugin (which adds extra metadata, like credits, to the displayed extended description).
I've pointed (https://www.beyonwiz.com.au/forum/viewtopic.php?f=55&t=9114&p=190703#p190703) adoxa (the plugin's author) to this discussion.
A new version of the plugin with the updated IceTV Genre tables has been merged into the source repository and should appear in the next release. Look for plugin version (in the MENU>IceTV heading) 20201026.
In the meantime, IceTV has started using a second new genre that's not handled by the current plugin (version 20191013), "History". Until the new plugin is available, expect to see logging like:
{3718}< 1839.374> [EPGFetcher] ERROR: lookup of 0x01 "Politics" returned "Cult"
{3718}< 1840.008> [EPGFetcher] ERROR: lookup of 0x01 "History" returned "Cult"
in the debug logs.
Note that when error messages like that indicate that there is a genre mapping failure in the plugin, the problematic genre in not added to the EPG (and not included in any recordings made of the affected program), not that the wrong genre is added to the EPG (and to affected recordings).
There are still some IceTV users that report problems with their Beyonwiz machines.
Has anyone at IceTV figured out what went wrong? What was the nature of the data being sent that caused the denial of service condition on the Beyonwiz machines? I'd like to put in a proper fix so that this does not keep on re-occurring.
I don't like the idea of shrugging the shoulders and saying it was fixed by turning it off and on again. It would be good to know if this is an inherent protocol/API issue, data validation issue, algorithm selection issue or just a plain old bug that needs specific conditions to trigger.
Peter, I cannot be of assistance. I am not aware of this happening with any of my T2s or my U4. I have not looked at the logs. If it will help I will do so.
Ian
Peter,
We do not know of anyone who still has the issue. If you do maybe you could illuminate us as to who so we can follow up? At the time of the alert Daniel and prl spent about three hours tearing out their respective hair but could not find any reason for the problem. In the end an API restart was tried and solved it but in doing that we lost any data to continue our examination. We did not shrug our shoulders and ignore it but commercially we can't tie up resources trying to find an obscure bug that has never appeared before and may not again. If it re-appears we will be a little more forewarned and will take appropriate action at the time.
Quote from: MD on October 30, 2020, 12:31:03 PMWe do not know of anyone who still has the issue. If you do maybe you could illuminate us as to who so we can follow up?
IanL-S reported instability yesterday and was looking to configure his systems to regularly reboot as a workaround for issues.
One of the problems here is that there is no obvious correlation between the symptoms that users experience and IceTV. The vast majority of users will not look at logs or suspect the IceTV plugin. The symptoms vary. Because the machine runs out of resources, failures can occur in any part of the code, resulting in unpredictable problems. Often these problems do not generate any logs. This lead a number of users to suspect hardware issues.
Quote from: MD on October 30, 2020, 12:31:03 PMWe did not shrug our shoulders and ignore it
I was not suggesting you did. I mean that I don't want to dismiss this as a non-issue for the Beyonwiz plugin code. I'd like to find the root cause and make the Beyonwiz plugin code more resilient. However, that is not possible without any clues as to what may have gone wrong. Some samples of the data that caused the issues would probably go a long way towards narrowing down the problem. Even server log information may provide some clues - were the responses larger than usual?
Hi, Colin. I haven't heard of any more reports of the "Error trying to fetch: Python int too large to convert to C long" problem since Daniel restarted the server at about 17:00 on Sun 25 Oct.
On Sunday, Daniel restarted the IceTV server while I was trying to find the cause of the "int too large" problem. I understand that he had a pressing need to get things working again, but it meant that I didn't get any useful debug information about the problem.
There have been a number of reports of problems with IceTV causing UI lockups on Beyonwizes, but there is only one that has any level of detail, from csutak40 (https://www.beyonwiz.com.au/forum/viewtopic.php?f=50&t=14488) (Judy). It shows evidence of possible out-of-memory issues, but none of them during the times on 24/5 Oct when the IceTV server was known to be causing the "int too large" problems. Some of csutak40's recent crashes were due to her T2's HDD filling, and those don't appear to be associated with IceTV in any way.
Other users report the UI (often of T2s) seizing up for no apparent reason, but so far, csutak40 is the only person to have provided any potentially useful debug information, but unfortunately there was little in that that is much help in localising the problem.
Ianl-S's post (https://beyonwiz.com.au/forum/viewtopic.php?f=46&t=14491) about his recent problems has really nothing in it that clearly either implicates or absolves the IceTV plugin, or its interaction with the server.
Grumpy_Geoff and I have both tried to replicate the problem, he on his in-use T2 and I on my test T2, but without any success.
Peter, thanks for the comprehensive update. Unfortunately I do not pay much attention to my T2s. In particular I was not paying any attention to the content of the IceTV log. Hopefully I will remember to check it out next time I have a problem.
Ian
Quote from: MD on October 30, 2020, 12:31:03 PM
We do not know of anyone who still has the issue. If you do maybe you could illuminate us as to who so we can follow up?
I only just found out that the user maindee reported issues on Sunday morning. This was in a post on the Beyonwiz forums https://www.beyonwiz.com.au/forum/viewtopic.php?p=190805#p190805 . There was no concrete evidence to link the issue to the IceTV plugin, but the reported symptoms match those reported previously.
I'm not sure if the above helps you to examine the back end and see if there is anything unusual there. The user "fixed" the problem by turning the machine off and on again, so there isn't much evidence available. This would have been sometime between 8:30am and 10:00am, 1 November 2020.