Question about seed.json and badger-sett

[tavitenari]

Hello, and thank you in advance for any insight. I'm hoping to get some clarification on the pre-learned heuristic data that comes with Privacy Badger.

It appears to me that the seed.json list is fetched from here during the build of the extension (via the Makefile), and then may be altered by the user if they choose to enable local learning. Am I correct in thinking that updates to the learned-list aren't continually fetched throughout the lifetime of the extension (ala the cookie yellowlist)?

Assuming I'm correct, are there plans to sync push pre-trained data to currently installed extensions? As it stands I believe users need to always install the latest version of the extension to get the latest heuristic data. (Apologies if I'm mistaken about this.)

[ghostwords]

Hello! Pre-trained lists are reloaded on Privacy Badger updates for users with default settings (local learning disabled).

As we wrote in the disabling local learning by default blog post, "users who do not opt in to local learning [...] will continue receiving information about new trackers we discover, keeping their Badgers up-to-date". We haven't yet implemented continually combining tracker lists with new data from Badger Sett for users who have enabled local learning, however. (If you have local learning enabled, your Badger can discover new trackers for itself.)

Does this answer your questions?

[tavitenari]

Ah yes, I didn't see that update function, that makes total sense. Thank you for explaining that, and yes it answers my question!

[tavitenari]

Does the data that comes from Badger-sett get vetted in some way before an update is pushed to privacy badger? It looks like the badger-sett list gets update every few days or so, but updates to seed.data in privacy badger get pushed in releases that are spaced out quite a bit further. Hypothetically, should it be safe/optimal to periodically pull seeddata direct from badger-sett, maybe even daily?

[ghostwords]

We review Badger Sett data for problems such as obvious site breakages that we should fix before releasing the data with the update.

We've also been steadily expanding the scope of crawls. The most recent Privacy Badger update comes with "synthetic" seed data generated from the combination of several recent 5,000 (and one 10,000) site crawls. We're looking into expanding the scope further, including improving our global coverage.

We may look into more frequent tracker updates in the future (by decoupling tracker data updates from our approximately monthly Privacy Badger releases), but I don't see much of a need for more frequent updates for now. We should instead focus on continuing to improve coverage, both through Community Learning (#1299) and larger/more global crawls.