[Bug] Apps frozen using pm hide can't be backed up

[devnoname120]

Context

Some battery-saving apps such as Hail offer to automatically freeze apps in order to prevent them from running background services that drain the battery. It can also create a shortcut for each application so that they can be transparently launched without manually unfreezing them.

Among the popular methods, there is pm disable, am force-stop, and suspend/pause). These three all have problematic limitations: received broadcasts and/or services and/or activities can still run in the background!

Then there is a fourth (incorrectly-named) solution: pm hide (formerly called pm block)

This very similar to uninstalling an app, but it doesn't actually remove the data and the package file, and the application still appears in Settings → Apps → See all 142 apps in Android. In this state an app cannot run any service, activities, intents, receive any broadcasts, etc.. It has a massive impact on battery life. I have around 60 apps frozen this way and my phone is as fast as stock.

Description

Neo-Backup doesn't display and thus cannot back up packages that were frozen using pm hide.

Currently Neo-Backup passes 0 for the flags argument of getInstalledPackages():

Neo-Backup/app/src/main/java/com/machiav3lli/backup/utils/ItemUtils.kt

Line 61 in bfc895b

getInstalledPackages(0).mapNotNull {

In comparison, Hail instead passes 8192:

aistra0528/Hail/app/src/main/kotlin/com/aistra/hail/utils/HPackages.kt (line 12-16)

    fun getInstalledApplications(flags: Int = if (HTarget.N) PackageManager.MATCH_UNINSTALLED_PACKAGES else 8192): List<ApplicationInfo> =
        if (HTarget.T) app.packageManager.getInstalledApplications(
            PackageManager.ApplicationInfoFlags.of(flags.toLong())
        )
        else app.packageManager.getInstalledApplications(flags)

This is probably the reason why Neo-Backup doesn't see the “hidden” apps even though they appear on both Hail and the list of apps in the Android settings.

Note that you can check whether a given app is in the “hidden” state or not by accessing its private flags.

Steps To Reproduce

1. Install Kiwi Browser, but any other app will work.
2. Open Neo-Backup and confirm that Kiwi Browser indeed appears in the list of apps that can be backed up. You can close Neo-Backup now.
3. Run adb shell pm hide com.kiwibrowser.browser, or alternatively freeze the app with Hail using the Hide freezing mode.
4. On Android, go to Settings → Apps → See all 142 apps → Kiwi Browser.
5. Confirm that you can indeed see the app details (there is even an Uninstall button).
6. Open Neo-Backup again (refresh the list if needed), and confirm that Kiwi Browser doesn't appear anymore in the list of apps. It's impossible to back it up with Neo-Backup now.

Expected behavior

Kiwi Browser still shows up in the list of apps in Neo-Backup and can be backed up without issues.

Screenshots

On the left Android shows Kiwi Browser in the list of apps and I can click on it to see the details, which enables me to uninstall it for example. On the right Neo-Backup doesn't show Kiwi Browser in the list, acting as if it were completely uninstalled.

System Information

• Device: Xiaomi Mi Note 10 Lite
• Android Version: 14
• ROM: PixelOS (PixelOS_toco-14.0-20231223-2156)
• App's Version: 8.3.5

[hg42]

I first tried with 8.3.5, I could not reproduce the issue with my first test app.
I used "Abfall App ZEW", because it's one of my first apps on the screen.

Note, now I'm on 8.3.6 (which shouldn't be different according to this issue).
With 8.3.6. it was the same...

Interestingly it's different for Kiwi browser, Adaway, Advanced Tools (didn't try more, yet, because this experiment removed the icons from my Nova Launcher).

Btw. you need to be root to execute pm hide, I assume you used adb root?

% adb shell pm hide com.kiwibrowser.browser

Exception occurred while executing 'hide':
java.lang.SecurityException: Neither user 2000 nor current process has android.permission.MANAGE_USERS.
...

this works (note, adbsu is a script that does echo "$*" | adb $device shell su, managing my various devices and emulators):

% adbsu pm hide com.kiwibrowser.browser 
Package com.kiwibrowser.browser new hidden state: true

% adbsu pm hide de.regio.abfallapp.zew
Package de.regio.abfallapp.zew new hidden state: true

alternatively freeze the app with Hail using the Hide freezing mode

which variant? there are multiple "Hide" options, I used "SuperUser Hide". This didn't change the state of "Abfall App ZEW".

With Hail, the icons are kept (at least some time), Nova still shows the icon, reporting "not installed" when clicking on it.
However, after some time the icon is gone. So it seems it was kept in a cache.

Interesting, that Hail can hide "Abfall App ZEW", while neither pm hide nor pm unhide works with Abfall App ZEW. The command always reports hidden state: true:

% adbsu pm hide de.regio.abfallapp.zew 
Package de.regio.abfallapp.zew new hidden state: true
                                                                                                                                                                  
% adbsu pm unhide de.regio.abfallapp.zew
Package de.regio.abfallapp.zew new hidden state: true

and that's not time based.

Summary: not all apps may react and there is a delay for certain following effects.

[hg42]

something like this is schizophrenic...

we might get certain problems, because the app is officially "not installed" and we might get errors.

• how do we detect that an app is hidden by pm hide?
  sorry, you already mentioned it:

Note that you can check whether a given app is in the “hidden” state or not by accessing its private flags.

• which of the functions and commands we use, will not work when an app is hidden?
• how can we work around that?

hopefully, it's a state that doesn't effect much for a superuser.

[hg42]

• what is this 8192 (a single bit, hex 2000), can we find a symbolic constant, that makes this future proof?

[hg42]

@machiav3lli

I thought it would be a simple change, but it's more complicated and I have not enough spare time for a longer trip:

• we use cached info, so applicationInfo must be handled in Package
• hidden is similar to disabled, but I'm not sure if some of the is* should be changed to a more live behavior
• isHidden needs a secret flag (the same applies to the "8192").
  I think we had a discussion some time ago, but don't remember the outcome.
  Do you want features based on non-official methods?
• PackageInfo is a database item, so if handled like disabled, this would create a new database version

[hg42]

I also tested using the 8192 in getInstalledPackages, but this doesn't help much, probably because isInstalled is still "no"
isInstalled -> isNotReallyUninstalled

[devnoname120]

@hg42

Nova still shows the icon, reporting "not installed" when clicking on it.
However, after some time the icon is gone. So it seems it was kept in a cache.

I don't have (and never had) that problem with KISS launcher. I guess it's a bug on Nova's side. Maybe it does aggressive caching and/or mishandles some broadcast events.

which variant? there are multiple "Hide" options, I used "SuperUser Hide". This didn't change the state of "Abfall App ZEW".

Interesting, that Hail can hide "Abfall App ZEW", while neither pm hide nor pm unhide works with Abfall App ZEW

Wait, so at the end is Hail able to hide "Abfall App ZEW" on your side or not?

I tried the methods Shizuku - Hide and Superuser - Hide and they both worked just fine on that app (and on my other apps as well). When it's hidden the app doesn't show anymore in Neo-Backup, and when it's unhidden it immediately reappears in Neo-Backup.

Note that Shizuku needs to be installed (+ granted root) for Shizuku - Hide to work, but it's way faster than the superuser method because Shizuku keeps a root service running instead of spinning su lots of times.

Btw. you need to be root to execute pm hide, I assume you used adb root?

I didn't try this method, it's what people recommend on the internet. I can try later if you want me to.

which of the functions and commands we use, will not work when an app is hidden?

Good question. I'm not entirely sure but since Android properly lists the hidden apps in the settings and it knows their state as well (some buttons are grayed out cf my screenshot), then I guess it's not too shaky either. What kind of commands does Neo-Backup need to run? I'd assume it mostly copies data directories and apks when backing up applications. Is there something else?

Summary: not all apps may react and there is a delay for certain following effects.

Did you try to force stop the app first? I haven't dug into Hail's source code but the answers are probably there. At least on my side I haven't had any issues or edge cases with Hail even though I've been intensively using it for the past 2 weeks.

what is this 8192 (a single bit, hex 2000), can we find a symbolic constant, that makes this future proof?

I'm not sure, but we may want to look it up on the internet. This private flag looks stable enough that Hail doesn't seem to have problems with this flag across the many devices and ROMs that people use (as far as I know).

hidden is similar to disabled, but I'm not sure if some of the is* should be changed to a more live behavior

I'm not sure to understand what you mean here. Could you expand?

PackageInfo is a database item, so if handled like disabled, this would create a new database version

Note that I'm not sure that an app can be in both states (disabled and hidden) at once. To be confirmed.

[hg42]

Wait, so at the end is Hail able to hide "Abfall App ZEW" on your side or not?

yes, and pm hide sometimes works, too, but most of the time not... I don't see a pattern

it feels like Hail does something different from pm hide

[hg42]

I don't have (and never had) that problem with KISS launcher.

does it remove the icon? or is it kept?

the interesting thing is: I restored the Nova data on my current device, and still have icons that are grayed out, because the apps are not installed (or not restored), yet. I guess deinstalling removes the icon. And pm hide seems to count as uninstall. NB and Nove both react on the action, NB replaces the icon by a generic icon. NB uses a broadcast to receive install/uninstall actions, I guess Nova does the same.

[hg42]

the app doesn't show anymore in Neo-Backup

yes, but only if there is no backup, otherwise the icon changes to a generic icon

[hg42]

which of the functions and commands we use, will not work when an app is hidden?

Good question. I'm not entirely sure but since Android properly lists the hidden apps in the settings and it knows their state as well (some buttons are grayed out cf my screenshot), then I guess it's not too shaky either. What kind of commands does Neo-Backup need to run? I'd assume it mostly copies data directories and apks when backing up applications. Is there something else?

well, I didn't think much about it, it's only from experience with such things...
For certain parts the hidden app is handled just like uninstalled, while it is not really uninstalled.
Such situation scream for problems...

Internal Anroid parts probably use states that are available to the system.
But it's not guarantied, that these states are available for apps. Apps generally see a facade. That you need to use reflection to access private flags is a good indication, that it's not ment to be used by apps.

Well, root applications are already on that path...so I wouldn't worry about it too much.
But it's a maintainance problem at the end. Hidden features will probably change more often.
The fact that there already are several ways to "disable" an app, shows that it's still in flux. They probably created this for their internal usage.

[hg42]

what is this 8192 (a single bit, hex 2000), can we find a symbolic constant, that makes this future proof?
[...]This private flag looks stable enough that Hail doesn't seem to have problems with this flag across the many devices and ROMs that people use (as far as I know).

well, for Hail it's the main functionality, it has no chance to ignore it...
but for NB it's an additional thing
Though I agree, that it would be nice to handle it.
Another possibility could be to unhide first, then hide again.
Btw. I think such states should be restored, too.

hidden is similar to disabled, but I'm not sure if some of the is* should be changed to a more live behavior

this was meant for Antonios (@machiav3lli)
isDisabled or isSystem are determined when the package list is created and updated in certain sitautions (e.g. install or uninstall events).
Hail does it live, asking the package manager on every invocation of isHidden.
This has advantages if you don't catch all possible (also future) events that could change the value.

PackageInfo is a database item, so if handled like disabled, this would create a new database version
Note that I'm not sure that an app can be in both states (disabled and hidden) at once. To be confirmed.

I can hide an app then disable it and then unhide, the result is a disabled app (you see it in NB, the disable buitton changes to enable, at the same time you cannot change disabled state in NB while it is hidden, not sure what is happening).

On the other side not all parts of the system seem to handle that situation.
E.g. pm list packages -d does not list a hidden app, even if it is disabled.
And there is no pm list packages -h.

That's the kind of things that I mean with "schizophrenic situations creating unexpected problems".
Even "Android" (or the developer maintaining pm) does not handle it completely and correct.
That's because it's added half hearted and should be considered as tinkering.

[hg42]

I used the original code from Hail to implement a test (cannot be kept as final code, because it breaks the concept).

I added isHidden and made isInstalled report hidden packages as installed.

Then the app sheet shows alle buttons but "Launch", which is good.
I can also create a backup... but, interestingly, it only creates a backup of the external data, no apk nor internal data nor de data.
I guess this part doesn't get the info it needs from the package manager.

--- mom ---

well, app and data sizes are zero in the app sheet

[hg42]

8192 is PackageManager.MATCH_UNINSTALLED_PACKAGES (which also matches installed packages),
but the symbol is only used for Android N, otherwise 8192 is used instead...
but why? the flag still exists...I would expect the opposite

[hg42]

adding that to getPackageStorageStats (only when retrieving the storageUuid) doesn't help, maybe because queryStatsForPackage only works for installed packages.

There are other cases, where storageStats would be interesting, e.g. uninstalled packages with data kept.

[devnoname120]

yes, and pm hide sometimes works, too, but most of the time not... I don't see a pattern

it feels like Hail does something different from pm hide

Here is the function that dispatches the freeze operation to the right function depending on the chosen mode:
https://github.com/aistra0528/Hail/blob/e030338611ad89c3c23d0ebb36cd4058dff900b5/app/src/main/kotlin/com/aistra/hail/app/AppManager.kt#L52-L67

fun setAppFrozen(packageName: String, frozen: Boolean): Boolean =
    packageName != BuildConfig.APPLICATION_ID && when (HailData.workingMode) {
        HailData.MODE_OWNER_HIDE -> HPolicy.setAppHidden(packageName, frozen)
        HailData.MODE_OWNER_SUSPEND -> HPolicy.setAppSuspended(packageName, frozen)
        HailData.MODE_DHIZUKU_HIDE -> HDhizuku.setAppHidden(packageName, frozen)
        HailData.MODE_DHIZUKU_SUSPEND -> HDhizuku.setAppSuspended(packageName, frozen)
        HailData.MODE_SU_DISABLE -> HShell.setAppDisabled(packageName, frozen)
        HailData.MODE_SU_HIDE -> HShell.setAppHidden(packageName, frozen)
        HailData.MODE_SU_SUSPEND -> HShell.setAppSuspended(packageName, frozen)
        HailData.MODE_SHIZUKU_DISABLE -> HShizuku.setAppDisabled(packageName, frozen)
        HailData.MODE_SHIZUKU_HIDE -> HShizuku.setAppHidden(packageName, frozen)
        HailData.MODE_SHIZUKU_SUSPEND -> HShizuku.setAppSuspended(packageName, frozen)
        HailData.MODE_ISLAND_HIDE -> HIsland.setAppHidden(packageName, frozen)
        HailData.MODE_ISLAND_SUSPEND -> HIsland.setAppSuspended(packageName, frozen)
        else -> false
    }

Here is what Hail does for the Superuser - Hide mode:
https://github.com/aistra0528/Hail/blob/e030338611ad89c3c23d0ebb36cd4058dff900b5/app/src/main/kotlin/com/aistra/hail/utils/HShell.kt#L24-L25

fun setAppHidden(packageName: String, hidden: Boolean): Boolean =
  execSU("pm ${if (hidden) "hide" else "unhide"} $packageName").first == 0

And for the Shizuku - Hide mode:
https://github.com/aistra0528/Hail/blob/e030338611ad89c3c23d0ebb36cd4058dff900b5/app/src/main/kotlin/com/aistra/hail/utils/HShizuku.kt#L90-L105

fun setAppHidden(packageName: String, hidden: Boolean): Boolean {
    HPackages.getApplicationInfoOrNull(packageName) ?: return false
    if (hidden) forceStopApp(packageName)
    return runCatching {
        val pm = asInterface("android.content.pm.IPackageManager", "package")
        pm::class.java.getMethod(
            "setApplicationHiddenSettingAsUser",
            String::class.java,
            Boolean::class.java,
            Int::class.java
        ).invoke(pm, packageName, hidden, userId) as Boolean
    }.getOrElse {
        HLog.e(it)
        false
    }
}

Interestingly for the Shizuku - Hide method it calls forceStopApp() before hiding the application, which is just a wrapper for ActivityManager.forceStopPackage().

Did you attempt to force stop the package before attempting to hide it? Maybe your issue is just that the pm hide command on your side doesn't work if the package is running.

does it remove the icon? or is it kept?

It removes it immediately (as it should). No latency, it's immediate and the same goes when unfreezing an app. I plan to hopefully do a PR to add support for hidden apps in KISS. It currently supports “hibernating” apps, but it's just an am force-stop under-the-hood.

Note that their documentation claims that “Hibernated app do not consume any battery or memory until you launch them again” but it doesn't match my experience: this command is pretty much useless because apps just restart at the next broadcast or intent.

NB and Nove both react on the action, NB replaces the icon by a generic icon. NB uses a broadcast to receive install/uninstall actions, I guess Nova does the same.

Sounds good to me so far. Pretending that the package was uninstalled from the point of view of user apps is the goal of hide.

the app doesn't show anymore in Neo-Backup

yes, but only if there is no backup, otherwise the icon changes to a generic icon

It seems to match the behavior of uninstalled apps, which I guess boils down to the fact that Neo-Backup thinks that they are actually uninstalled because it doesn't pass the 8192 flag to packageManager.getInstalledApplications().

For certain parts the hidden app is handled just like uninstalled, while it is not really uninstalled.
Such situation scream for problems...

I agree, but on the other hand silently ignoring these apps for backup operations is dangerous IMO. I wouldn't be too surprised if some manufacturer ROMs started using this hide feature for battery-saving purposes (if they don't already). OxygeneOS (formerly MIUI) for example is known to be extremely aggressive when it comes to battery-saving operations, and I don't think that they would leave such a battery life-improving opportunity on the sidewalk.

That you need to use reflection to access private flags is a good indication, that it's not ment to be used by apps.

I agree. The goal here for Android is to pretend that the app was entirely uninstalled, because it makes sure that it doesn't break anything from the point of view of other apps. But under-the-hood it keeps its package and data, and pretends that it just got installed again when the unhide operation is performed. This abstraction transparently makes it work with everything without breaking backward compatibility — they need not even be aware of that OS feature at all.

Another possibility could be to unhide first, then hide again.
Btw. I think such states should be restored, too.

This could definitely be an option. I think it would be a bit costly for the battery if you use the su method though rather than Shizuku. I'm not sure about the exact impact of it, it may be marginal enough since it's just a few seconds. You don't want Neo-Backup to crash in the middle and leave some apps unhidden though.

On the other side not all parts of the system seem to handle that situation.
E.g. pm list packages -d does not list a hidden app, even if it is disabled.
And there is no pm list packages -h.

As far as I understand it's the expected behavior. pm list packages -d shows packages that are installed but disabled. Why should it show packages that it pretends are uninstalled for all intents and purposes? Again, this is important for backward-compatibility that this feature doesn't break or change anything.

8192 is PackageManager.MATCH_UNINSTALLED_PACKAGES (which also matches installed packages),
but the symbol is only used for Android N, otherwise 8192 is used instead...
but why? the flag still exists...I would expect the opposite

Interesting… Are you certain that this constant indeed has the value 8192 in all newer Android versions? I guess we could ask Hail developers they probably have an answer as to why they did that.

[devnoname120]

@hg42 Friendly bump 🙂

[hg42]

the bump worked...

Well, I already have a branch that should list hidden apps, but I forgot, what the state of that patch is...
Did I provide a test apk = pumpkin apk on Telegram?

I also forgot, that I wrote parts of the test results here (which is natural, haha).
Too many projects going on here (I'm currently into music making, which is a totally different concept).

I agree, am force-stop

is pretty much useless because apps just restart at the next broadcast or intent

Some services even restart immediately.
Android would pause an app anyways if it needs memory. And most apps are put to sleep nowadays.

I guess, the only thing, such an app should do, is restricting background work, that is de-register alarms, broadcasts etc. (which hide/disable/suspend etc. do). Actually, it should do this only for unnecessary work, like downloading advertisements etc.
Though it is questionable how to do that automatically (detect the unnecessary things).

Btw. it seems that pm suspend is a better way than hide? does it have disadvantages?
At least the launcher doesn't remove the icon, while with disable and hide it disappears (I think -- I'm a Nova user).
I'm not sure, but hide seems to be an uninstall only for the current user? at least the unhide can also be done with install + user as far as I remember.

At the moment, I think,

it doesn't pass the 8192 flag to packageManager.getInstalledApplications()

is the correct (or consequent) behavior, because NB currently only works for other current user and hide uninstalls the app for that user.

When NB starts supporting work profile and multiuser when running on the main user, the condition would change.
The hide is a kind of tricking the system. Well, that does not necessarily mean, that NB should not make this possible.
But the architecture will probably change and the current work may be useless.

You don't want Neo-Backup to crash in the middle and leave some apps unhidden though

this should be the same with commands or API calls

pm list packages -d shows packages that are installed but disabled. Why should it show packages that it pretends are uninstalled for all intents and purposes

I think, disable and hide are very similar. At least disable is also like uninstalled for all intents and purposes.
Actually, I could not say any difference from the POV of the current user, can you?
I think the difference is in the implementation, and the effect seems to be the same. I think hide was created for multiuser/profiles etc.
As far as I remember hide doesn't influence other users/profiles, but I guess, disable does it for all profiles.

Are you certain that this constant indeed has the value 8192 in all newer Android versions

no, I only looked at the current SDK and Android N.
But it doesn't make any sense to change the CONSTANT and keep the value for the same purpose?
Usually the value would come first and the symbol added later. I could only think of a deprecation or maybe removing the symbol from the usual developer namespace, and providing it only internally to Android sources.