Topics

Comskip is used for detecting and skipping ads and as far as I can tell, the current comskip configuration produces a number of files.  The 3 files of interest are: Recording.edl, Recording.txt and Recording.vdr.

When Kodi loads a recording to be played, it requests the 'cutpoints' from TVHeadend.

TVHeadend will first try to read the cutpoints from Recording.txt.  If it is successful, it will stop there.  If no Recording.txt file is found, TVH will then try to read the cutpoints from Recording.edl.  Recording.vdr is never referenced.

I have been doing some work on TVH that will write 'scene marker' records into an 'edl' file for the start and then end of the EPG event.  If, for example, there are 2 minutes of pre-padding a 30 minute programme and then 5 minutes of post-padding, scene markers will be inserted at 2 minutes and 32 minutes.  These scene markers are written immediately after the recording has finished.

Unfortunately, when comskip runs, it overwrites the scene marker edl records that were just created.

I would like IceTV to consider making a few changes so that these two features can coexist.

comskip.ini

Code Select Expand

output_default=0    ; Turn off the default comskip 'txt' output.
edl_skip_field=3    ; Use the ad break marker type (3) instead of the cut marker type (0).
output_vdr=0        ; Turn off the unneeded 'vdr' output.

In addition to this, I have written small proof-of-concept script that demonstrates the principle of saving the TVH scene markers before comskip runs and then reapplying them once comskip has created its edl file.

Code Select Expand

#!/bin/bash
#Get the file name only
BASEONLY=`basename "$1" mkv`
#Get the directory name only
DIRONLY=`dirname "$1"`
#Create the EDL file name
FULLNAME="$DIRONLY/$BASEONLY"edl
#If the EDL file exists, save the scene marker records (type = 2)
if test -f $FULLNAME; then
    cat $FULLNAME | awk '$3 == 2 {print $0}' > "$FULLNAME"_saved
fi

#Run comskip
/usr/share/kodi/addons/service.tvheadendicetv/bin/comskip --ini=dmc_comskip.ini sample.mkv

#If a saved EDL file exists, add that to the end of the new EDL
if test -f "$FULLNAME"_saved; then
    cat "$FULLNAME"_saved >> $FULLNAME
    rm "$FULLNAME"_saved
fi

With a combination of these two features, the user will be able to jump to near the beginning of the programme and then also skip the ads during playback.  Here is what the final result looks like, with annotations.



Thoughts and comments welcome.

If anyone is interested in getting FTA EPG for the Radio channels, I can post some detailed instructions.



Please note:  This will likely void your warranty.

I'm not sure if this is a known issue, or indeed how many users this could impact.

I have set my test/dev IceBox to prepend the repeat flag to the title field.

This causes a timer creation to fail.  Here is an excerpt from my log:

Code Select Expand

Title is not matching, do not schedule the show
Got [[R-DEV] Hard Quiz] but was expecting [Hard Quiz]
Show:
{'eventId': 1165, 'channelName': 'ABC TV', 'channelUuid': 'c80811a5c49a725d0bb2f3e6eb447f39', 'channelNumber': '0', 'channelIcon': 'imagecache/2', 'start': 1706686200, 'stop': 1706688000, 'title': '[R-DEV] Hard Quiz', 'subtitle': 'Series 8, Episode 18', 'description': 'Tom Gleeson is hanging on to the Big Brass Mug until one Hard Quiz expert can prove they deserve it - but with which topic? Australian parrots, cricket bats, bones, or WikiLeaks?', 'repeat': 1, 'subtitled': 1, 'image': 'http://images.icetv.com.au/17a6-dfb1-1fa9-d0e4.jpg', 'ageRating': 7, 'first_aired': 1672531680, 'copyright_year': 2023, 'nextEventId': 1166}
Timer:
{'id': '1706682258', 'device_id': 188516, 'channel_id': 2, 'name': 'Hard Quiz', 'show_id': 180966489, 'start_time': '2024-01-31T07:30:00+00:00', 'duration_minutes': 30, 'action': 'record', 'state': 'waiting', 'message': 'Created with Interactive web interface.', 'series_recording_id': 0, 'keyword_id': 0, 'rescheduled_by_tca': False}
Could not get event ID
Missing information from tvheadend, can't proceed
No timer changes, local cache was not updated
End timer update
I have not tested appending the repeat flag, however, I suspect that the result will be the same.

Matching the names is probably the best that can be achieved because AFAICR the XMLTV specification does not support an eventID field (something that I am looking into for TVH).  Perhaps that match could be a substring search rather than strict equivalence.

[1)]

1) I was experimenting with the Ice TV Kodi addon over the weekend and I found that regardless of the host name that I provided during the LibreELEC portion of the setup process, the device was always named 'IceBox' under 'My Recorders' on the ICE web site.

After a quick search, I found that in 'icetv.py' in the 'login_via_setup_wizard()' function, the device label seems to be hard-coded rather than using the Linux host name (uname -n).

Code Select Expand

    if (region > 0):
        data = {'member': {'email_address': email_address,
                           'password': password,
                           'region_id': region},
                'device': {'uid': mac_address,
                           'label': 'IceBox',   <====== Hard-coded
                           'type_id': 40}}
    else:
        data = {'member': {'email_address': email_address,
                           'password': password},
                'device': {'uid': mac_address,
                           'label': 'IceBox',   <====== Hard-coded
                           'type_id': 40}}
Perhaps this is not the right area of the code, however, there are a number of Python calls that can get the hostname.


2) I also noticed that the addon is hard-coded to use 'service.tvheadend42'.  LibreELEC are currently working on 'tvheadend43'.  It might be handy to search for the loaded Kodi services and match any version of TVH.


3) Are there any plans to release this addon (or a variation thereof) as a generic Kodi-installable addon?

I would like to investigate paying via direct credit.  Are the following instructions still valid?

https://support.icetv.com.au/hc/en-us/articles/227082547-Can-I-renew-my-subscription-by-direct-deposit-

Has there been any thought about incorporating IceTv account management into the IceBox?  Things like creating/editing/deleting searches and favourites and getting recommendations, etc.

The last time I checked, timers created on the IceBox were not reflected on the Ice web site.  Is this still the case?  Synchronising timers would be good to see.

Instead of a weekly email, perhaps Kodi could deliver a 'What's new on ICE this week' screen that would have links to EPG and the ability to set timers.

With LibreELEC/IceBox, tvheadend is not provided as a stand-alone package, but as a Kodi addon that launches the tvheadend executable on startup.  This Kodi Addon reports its version number as being 4.2.8-36.

The WebUI reflects this as well stating: Build: 4.2.8-36 ~ LibreELEC Tvh-addon v11.0.0.0 (2023-04-26T23:18:34+0000)

On the TVH GitHub repository, however, the latest published version seems to be 4.2.7.  Furthermore, the last release of TVH seems to have been over 5 years ago.

It would seem that perhaps LibreELEC build their own TVH for inclusion with the Kodi distribution that they bundle.  The higher version number plus the suffix plus the build date suggest that LibreELEC may also have done some patching and/or improving somewhere along the way.

My question for IceTV is:  Do you rebuild LibreELEC from source or do you take a pre-built LibreELEC image and rebrand it?

If the IceBox 'firmware' is built from source every time, then the possibility of modifying TVH in a meaningful timeframe is much greater.

I was wondering if IceTV could issue us with a bit of a roadmap.

• What bugs/issues are being tracked and what are the current statuses?
• What new features are being considered/under development?
• When is the next release of the Kodi addon expected and what updates will it contain?

Stuff like that.

[adjusted]

I have noticed that the parental rating does not get into the TVH EPG database.  This is because TVH does not understand the 'ACMA' codes.

Code Select Expand

<rating system="ACMA">
<value>PG</value>
</rating>
In my investigation of the TVH source code, I discovered that it will also accept a raw number if it is provided and that number will be used as the parental rating value.

Code Select Expand

<rating system="ACMA">
<value>11</value>
</rating>
Normally, the EIT rating provided is the recommended age minus 3.  TVH, on the other hand, expects the actual recommended age to be provided via XMLTV.

Based on experiments long ago on a Topfield PVR feeding in EIT data, here are the codes and adjusted age values that it uses:

6 = "P"
7/8 = "C"
9/10 = "G"
11/12 = "PG"
13/14 = "M"
15/16 = "MA"
17 = "AV"
18 = "R"

The rules may have changes since then.

I wanted to investigate if I could have an alternate/backup EPG source in case my internet access was ever interrupted for an extended period.  The ICE EPG is great, however, if your internet goes down, then it is not updated.

I'm looking into this because many, many years ago, I came home after several weeks away to find that although my Topfield PVR had completed all of my timer-based recordings expected, they all lacked EPG information.  This was because during my absence, my ISP connection hardware had gone belly-up and my Topfield PVR could no longer talk to the ICE server.

My eventual goal is to set keyword searches in TVH that only use the FTA EPG and create a low-priority scheduled recording.  These recordings will be given a short-ish expiry date.  All of my 'normal' recordings will still be managed by Kodi/ICE.

Caveats:

• Doing this may void your warranty.
• This is only my recollection.
• I probably should have kept better notes.
• I may have forgotten a step or two.

This is what I have discovered so far.

• Created a new 'Network' called 'DVB-T Free-To-Air'.  This is so that 2 sets of muxes (TV frequencies to tune to) can be created, but with different EPG rules.
• Go into 'TV Adapters' and ensure that the new network is included in the 'Networks' list for all adapters.  This is so the new network can attach to a tuner when required.
• Created a new duplicate mux for every original ICE mux delivering services, but using the new network, not the default ICE network.  Tvheadend also has lists of pre-defined muxes for many locations.
• The new FTA muxes need 'EPG Scan' changed to 'Only EIT'.  This is so that they read the EPG from the FTA feed.
• The existing muxes created by ICE need 'EPG Scan' changed to 'Only Open TV Sky Ausat'.  This is to prevent them from using the FTA EPG, however, it does not seem to stop them from received the XMLTV EPG data that ICE sends.  Failure to do this will result in a mixture of ICE and FTA EPG data for channels on this mux.
• Once the new FTA muxes have been created, they will automatically scan for new services.
• Create a new 'Channel' for each of the new services found on the new muxes making sure to give them a name (I prepended 'FTA-') and/or LCN (I added 1,000) that will clearly identify them as having a separate EPG source.  Ensure that 'Automatically map EPG source' is enabled and that 'EPG source' is empty.  The 'EPG source' for the existing ICE channels should always have the XMLTV grabber set.
• Make sure that the 'Over-the-air EIT: DVB Grabber' is enabled in Configuration -> Channel / EPG.
• Ensure that the OTA Cron multi-line contains a line containing '*/15 * * * *'.  This will run an FTA EPG search every 15 minutes if a tuner is available.

If nothing else, this process may be handy for getting EPG information for 'radio' channels not provided by ICE.

I've done some testing over the previous months regarding getting the IceBox to wake using the RCU.  So far, I have had no success.  However, here is what I tried and what I found.  Perhaps someone else may have some other ideas.

Lets start with a positive - Timed Waking

The IceBox supports Linux rtcwake capabilities.  This means that the unit can be woken from a powered-off state in order to perform recordings.  I had to write directly to the RTC device in order for this to happen, but at least it is possible.  Perhaps Kodi can already utilise this feature, more investigation is required.

And now for the rest - Sleeping / waking, etc.

I have tested the IceTV remote with its USB dongle in a Windows 10 PC and a PC running Ubuntu.  On both systems, the ICE RCU was able to generate arrow keys, etc, and the power key was able to put those systems to sleep/hibernate.

Unfortunately, the IceTV RCU was unable to wake either system.  Both systems would wake on USB keyboard input and I even tried plugging the ICE dongle into the exact same port that a known good waking USB keyboard was in plugged into, however, the systems could not be woken with the ICE RCU.

Given that the IceBox can not be woken with the ICE RCU, perhaps a real USB keyboard can wake it.  Unfortunately not.  I have been unable to make a USB keyboard wake the IceBox.

Perhaps the IceBox can't be woken because it can't get to sleep in the first place.

Most modern computers can 'sleep' where memory is kept active but the rest of the computer is shut down and 'hibernate' where the memory is saved to disk before the unit is powered off.

I could get neither of these modes to work.

During my tests, I discovered that LibreELEC (the OS that the IceBox is based on) has 'swapping' disabled by default.  This feature is required for systems to be able to hibernate.

I followed the instruction and got a swapfile working, however, I was still unable to get the unit to hibernate.  Kodi would sort of freeze in a dimmed state and a full power reset was required to return the unit to operation.

I have noticed that a significant number of programmes are missing genre descriptions that should have them.



I checked on the IceTV web site, that the EPG clearly shows genre descriptions.



This is also reflected in the raw ICE EPG JSON feed.

Code Select Expand

        }, {
            "id": "176785103",
            "series_id": "9892",
            "episode_id": "216252",
            "channel_id": "38",
            "date": "2016",
            "season": "5",
            "episode-num": "4",
            "start": "2023-07-10T08:57:00+00:00",
            "stop": "2023-07-10T09:05:00+00:00",
            "title": "Shaun The Sheep",
            "subtitle": "Baa-d Hair Day",
            "desc": "The flock don't recognise Shaun when he loses one his most distinctive features. His old pal Bitzer helps him track down the lost thatch, but not before Shaun becomes very attached to one of the chickens",
            "icon": {
                "src": "http://images.icetv.com.au/ee37-9443-b346-38c3.jpg",
                "width": "831",
                "height": "467"
            },
            "category": [{
                    "name": "Children",
                    "eit": "0x50"
                }, {
                    "name": "Animation",
                    "eit": "0x55"
                }
            ],
            "language": "English",
            "country": "United Kingdom",
            "video": {
                "aspect": "16:9",
                "colour": "YES",
                "quality": "SDTV"
            },
            "previously-shown": {
                "start": "2016-01-13",
                "channel_id": ""
            },
            "subtitles": {
                "onscreen": "English"
            },
            "part_of_series": "Yes",
            "rating": "G",
            "external_ids": {
                "tvdb_series": "79890",
                "tvdb_episode": "5544592",
                "imdb_series": "tt0983983",
                "imdb_episode": "tt6226402"
            }
        }, {
I did some experimentation with XMLTV importing and I found that "Children's / Youth programs" was accepted, but that "Children" was not excepted.

Having a quick look at the tvheadend source code, it would seem that it is doing some text matching between the received genre description and the ETSI genre descriptions.

https://github.com/tvheadend/tvheadend/blob/14298acb6a8e3a83ed1091fab1f3a924077ddfea/src/epg.c#L1820C26-L1820C26

More investigation is obviously required, however, it would appear that IceTV needs to pass the ETSI description, or at least TVH's version of the ETSI description, for the genre description to be accepted.

The JSON feed already has the ETSI code, perhaps the ICE Kodi addon needs a lookup table to find a value that will work with TVH.

[Playing Recordings]

I have a number of Topfield PVRs and I have been experimenting with interoperability between Topfield and IceBox.  Here is a summary of my findings:

Playing Recordings

If you copy your Topfield recordings onto your Kodi/IceBox, they can be played without issue, however, the Topfield metadata in not available, even if you copy the INF file.

If you want to copy any Topfield recordings, you can do that via a USB flash drive, etc.

If, however, your Topfield PVR and IceBox are on the same network, you can go into the Kodi file manager and add a network source using FTP as the protocol.  This will enable you to copy the recordings directly from the Topfield to the IceBox.  I can add a step-by-step guide if anyone is interested.

Getting Metadata

To get metadata working, I firstly needed to copy the Topfield recordings into a Kodi 'Movie' library.  I wrote a script that will read the metadata from the Topfield INF file and create a Kodi 'NFO' file.  The Kodi movie library then needs to be rescanned and the metadata will appear.

The script is a once-off that needs to be run from the Linux command line on the IceBox to import the metadata.  If anyone is interested, I can make this available.

Playing recordings still on the Topfield PVR.

There are a set of TAPs for the Topfield called TMSServer and TMSClient.  These two TAPs work together to allow one Topfield PVR to play recordings residing on another Topfield PVR.  These TAPs use NFS as their data transport, but advertise via a proprietary protocol.

There is another TAP called WebControl that provides a JSON interface to the Topfield PVR and allows for control via a web browser and other systems that can read and process JSON data.

I have written an addon for Kodi that uses a combination of TMSServer advertisement broadcasts to learn what Topfield PVRs are available and WebControl to extract PVR-specific information.  I have called this addon 'TMSClient' and it is able to list the recordings on a Topfield PVR, with metadata, and play those recordings.  There is no copying involved, playback is achieved directly off the Topfield's HDD over the network.  The addon can also list the upcoming timers on the Topfield PVR read-only.

Timers

I have also written a proof-of-concept script that can copy all of the timers from a Topfield PVR onto the IceBox.  For now, every timer is copied as a once-off timer regardless of its frequency setting.  The script uses WebControl to extract configuration data from the Topfield so that it can match the DVB triplets of each channel to create the timers for the correct channels.

I have been investigating how tvheadend (TVH) stores its EPG data and from what I have found, TVH holds the EPG data in memory and only writes it to disk during a clean exit.  I need to do some more testing, however, I suspect that this means that any new EPG data received between system startup and a power failure will be lost.

For FTA EIT, this would not represent a huge problem because the EPG grabber task will run on startup and the EPG will be replenished.

For ICE, however, I believe that a power failure will result in a hole in the EPG data.  This hole will span all of the EPG data received between the last clean shutdown and the restart after the power failure.

As the ICE addon receives new EPG data, that data is processed, sent to TVH and then ICE saves its 'last update' time (I have confirmed that this is actually written to disk every minute).  For the next EPG cycle, ICE requests EPG changes since the 'last update', applies only those changes and the process repeats.

However, if a power failure has occurred, TVH will not have written any newly received EPG data to its database file.  ICE will not be aware of this and continue to request EPG data since its 'last update' received, not the last update retained by TVH.

I'd like to propose 2 possible resolutions:

1) ICE sets a flag in its local settings file on startup and then resets that flag when shutdown cleanly.  If the ICE addon starts and finds the flag set, there is a good chance that there was a power failure and it reloads the entire EPG into TVH.

2) ICE keeps track of the signature (eventid/channel/start/stop) of the last EPG entry successfully sent to TVH.  ICE also keeps track of the 'last update' value that the previous session started with.  On startup, ICE looks for the EPG entry signature and if it is not found, reloads the EPG from the 'last update' of the previous clean session.

Option 1 is probably easier to implement, however, it has a larger network impact for frequent power failures.  Option 2 has slightly more moving parts, but only re-requests the potentially missing data, not the whole EPG.

The one last thing that I have to investigate is whether the Kodi DVR function caches EPG information to disk, or if it obtains a fresh EPG data set via HTSP on startup.

If I have misinterpreted anything, I'm happy to be corrected.

One thing that was bugging me, compared to my old Topfields, is that my recordings are grouped by programme name and not presented chronologically.



I found that I could enter 'Options' and select 'Group Items' as false.



After that, the recordings appeared in chronological order and not grouped by programme.

I noticed an annoying (to me) dimming of the screen when the IceBox was left unattended.  I can't remember if this also happened with the first firmware that I had, but it became noticeable after the upgrade to the 22-May-2023 firmware, or most likely, Kodi 20.1.0.

I found that the screensaver had been (re?)activated and the default was 'Dim'.  I disabled this as follows:

Settings -> Interface -> Screensaver -> Screensaver mode = None

I just performed the upgrade to the 22 May 2023 firmware.  The update process seemed to go very smoothly (Thanks Ice guys!), however, when I went back into the guide, it was now showing sequence numbers for the channel numbers and not the LCNs as I had previously set it to do.

I went into the Kodi DVR settings and saw in my channels settings that it was still configured to show the 'channel numbers from backend' as I expected.

In typical IT support fashion, I turned this setting off, went into the guide, went back to the setting and turned it on again.  When I went back the guide once more the LCNs were back.

I'm guessing that there was an issue with the setting and turning it off and back on again resolved the issue.

I noticed that pre/post padding for timers can be specified in a number of places.  I did a little bit of research into this and here is a flowchart that I drew to explain my findings.



I like the fact that channels can get their own padding.  This means that we can set much wider margins for the commercial networks and keep the public-ish networks a little more modest.

Please let me know if you find any omissions or inaccuracies and I will update the flowchart.

As yesterday's announced firmware upgrade will be my first one, I'd like to know what data is lost as part of the process.

Recordings?
Timers?
Music?
Logos?
Settings?
Addons?
Other contents of /storage?

If anyone is interested in adding some channel logos to their IceBox, I have attached them in a zip file.  logos.zip



This is just a first pass proof-of-concept and for Sydney/TV (no radio) only.  However, some of the logo names seem to be generic so they should work in other cities.

The ABC/SBS logos look a bit dodgy at the moment.  I think that this is because they are monochrome and Kodi seems to invert the colours in some places.

You need to have FTP access to your IceBox or copy the files across from a USB flash drive, etc.

• The PNG files need to be placed into '/storage/picons/tvh' on your IceBox.
• Once this is done, you need to browse into tvheadend on your IceBox using 'http://<IceBox IP>:9981/'
• Go into Configuration|Channel/EPG.
• Select all of the channels with CRTL-A.
• Press the 'Reset Icon' button.
• Press the 'Save' button.
• Tvheadend will then scan for the logos and load the ones that it finds.

As stated, this is only a first pass proof-of-concept.  I wrote a script that gets a list of ICE channels for my IceBox's current region.  This list contains the DVB triplets for each channel plus the URL for the logo file.  The script then extracts the channel information from tvheadend so that it can match the DVB triplets, extract the display name for the channel and map the display name to the URL.