Get all Public Folders

[dtalkachou]

Previously we retrieved all public folders for an account by account.public_folders_root.root.children. But it doesn't work for about three weeks, it returns an empty generator. Even though account.public_foolders_root.child_folder_count returns the number that matches real public folder numbers, we still cannot find a way to iterate through all of the IPM folders as we did before. We use version 4.7.2 but don't mind switching if it helps. Does anyone can help us?

[ecederstrand]

I think this is connected to the issue reported in #1273.

I don't have access to a test account with public folders, so I would suggest to debug this as described in #1273 (comment)

Please note that it would help if you debug from commit f65079f which is still unreleased, since there were changes related to public folders in that commit.

[dtalkachou]

@ecederstrand I think it's somehow related because appeared about the same time, but not our problem. We work with public folders and can't get a list of all IPM folders. I also noticed that the code account.public_folders_root // "folder_name" works for public folders, but our first problem is getting a list of all IPM folders as I said. Previously we did it by account.public_folders_root.root.children, but now it provides an empty generator.

I tried the commit you specified, but it's still an empty generator.

[ecederstrand]

Ok. Adding a breakpoint in BaseFolder.children() and stepping through the code is your best bet in debugging this. If you enable debug logging you should be able to see the XML requests and responses while you are stepping through the code. That might give you an idea of where the problem lies.

[ecederstrand]

Actually, there's a slight possibility that d9035d0 fixes this. Can you give it a try?

[dtalkachou]

Thank you very much for your involvement in our problem, but it still doesn't solve it.

[ecederstrand]

@dtalkachou Did you find a solution to this? Is it apparent in all exchangelib versions, or is there a certain one that exhibits the error?

[dtalkachou]

No, I haven't. It's not dependent on the version. It returns an empty generator since somewhen.

[iglimanaj]

Hi @ecederstrand,
The problems emerged at the beginning of April. Back then we had version 4.7.2. After upgrading to 5.4.0 the problem is still present.
Interestingly, I found out that account.public_folders_root._folders_map.values() contains indeed the list of the first level children of public folders. However, .get_children() still returns an empty generator.

Any advice for further investigation would be very much appreciated!

[ecederstrand]

Hi @iglimanaj,

Thanks for the additional information! Interesting that _folders_map does contain the public folders. This must mean that the parent folder ID of the child folders does not match the ID of account.public_folders_root in

exchangelib/exchangelib/folders/roots.py

Line 101 in c310c65

if f.parent.id == folder.id:

You could add a breakpoint there and check what the child and parent folder ID's are in your case.

[iglimanaj]

Thanks for the hint @ecederstrand!
I realized that that line of code is not even reached because the loop is continued one line before, because not f.parent is resolved to True

exchangelib/exchangelib/folders/roots.py

Line 99 in c310c65

if not f.parent:

So, the problem must be somewhere else? 🤔

[ecederstrand]

Weird. We ask for the ParentFolderId field when getting subfolders. If you enable debug logging and check the request and response XML, you should be able to see whether we are requesting that field when we request subfolders of publicfoldersroot, and whether the subfolders have that XML element when they are returned.

[iglimanaj]

Yes, that field ist being requested and is also part of the response. f.parent_folder_id is also returning a value, even though f.parent is set to None

[ecederstrand]

Folder.parent is a property, defined here:

exchangelib/exchangelib/folders/base.py

Line 869 in c310c65

def parent(self):

It ends up calling RootOfHierarchy.get_folder(), defined here:

exchangelib/exchangelib/folders/roots.py

Line 72 in c310c65

def get_folder(self, folder):

which looks up by ID in _folders_map.

You'll either need to trace the code into those methods to see what's going on.

[iglimanaj]

I set the breakpoints according to your recommendations and realized that parent_folder_id is set to a value which cannot be found in _folders_map.

I get for example the following _folders_map

{'AQEuAAADGkRzkKpmEc2byACqAC/EWgMAh2nBrScK1Uu67im0P7ruQQAAAwIAAAA=': PublicFoldersRoot(<exchangelib.account.Account object at 0x105233290>, '[self]', 'IPM_SUBTREE', 0, 0, 5, None, 'AQEuAAADGkRzkKpmEc2byACqAC/EWgMAh2nBrScK1Uu67im0P7ruQQAAAwIAAAA=', 'AQAAABYAAACHacGtJwrVS7ruKbQ/uu5BAADeEghM'), 'AQEuAAADZjPaKB5hpEqkLxzShIQASGgBAIdpwa0nCtVLuu4ptD+67kEAAAMUAAAA': Messages(PublicFoldersRoot(<exchangelib.account.Account object at 0x105233290>, '[self]', 'IPM_SUBTREE', 0, 0, 5, None, 'AQEuAAADGkRzkKpmEc2byACqAC/EWgMAh2nBrScK1Uu67im0P7ruQQAAAwIAAAA=', 'AQAAABYAAACHacGtJwrVS7ruKbQ/uu5BAADeEghM'), 'public_folder_1', 2, 2, 0, 'IPF.Note', 'AQEuAAADZjPaKB5hpEqkLxzShIQASGgBAIdpwa0nCtVLuu4ptD+67kEAAAMUAAAA', 'AQAAABYAAACHacGtJwrVS7ruKbQ/uu5BAAAAAAZK'), 'AQEuAAADZjPaKB5hpEqkLxzShIQASGgBAIdpwa0nCtVLuu4ptD+67kEAAAMWAAAA': Messages(PublicFoldersRoot(<exchangelib.account.Account object at 0x105233290>, '[self]', 'IPM_SUBTREE', 0, 0, 5, None, 'AQEuAAADGkRzkKpmEc2byACqAC/EWgMAh2nBrScK1Uu67im0P7ruQQAAAwIAAAA=', 'AQAAABYAAACHacGtJwrVS7ruKbQ/uu5BAADeEghM'), 'public_folder_2', 2, 1, 3, 'IPF.Note', 'AQEuAAADZjPaKB5hpEqkLxzShIQASGgBAIdpwa0nCtVLuu4ptD+67kEAAAMWAAAA', 'AQAAABYAAACHacGtJwrVS7ruKbQ/uu5BAAAAAACU'), 'AQEuAAADZjPaKB5hpEqkLxzShIQASGgBAIdpwa0nCtVLuu4ptD+67kEAAAMYAAAA': Messages(PublicFoldersRoot(<exchangelib.account.Account object at 0x105233290>, '[self]', 'IPM_SUBTREE', 0, 0, 5, None, 'AQEuAAADGkRzkKpmEc2byACqAC/EWgMAh2nBrScK1Uu67im0P7ruQQAAAwIAAAA=', 'AQAAABYAAACHacGtJwrVS7ruKbQ/uu5BAADeEghM'), 'public_folder_3', 0, 0, 0, 'IPF.Note', 'AQEuAAADZjPaKB5hpEqkLxzShIQASGgBAIdpwa0nCtVLuu4ptD+67kEAAAMYAAAA', 'AQAAABYAAACHacGtJwrVS7ruKbQ/uu5BAAAAAACZ'), 'AAEuAAAAAABmM9ooHmGkSqQvHNKEhEhoAQCHacGtJwrVS7ruKbQ/uu5BAADd4OCcAAA=': Messages(PublicFoldersRoot(<exchangelib.account.Account object at 0x105233290>, '[self]', 'IPM_SUBTREE', 0, 0, 5, None, 'AQEuAAADGkRzkKpmEc2byACqAC/EWgMAh2nBrScK1Uu67im0P7ruQQAAAwIAAAA=', 'AQAAABYAAACHacGtJwrVS7ruKbQ/uu5BAADeEghM'), 'Public_folder_oleh', 3, 3, 0, 'IPF.Note', 'AAEuAAAAAABmM9ooHmGkSqQvHNKEhEhoAQCHacGtJwrVS7ruKbQ/uu5BAADd4OCcAAA=', 'AQAAABYAAACHacGtJwrVS7ruKbQ/uu5BAADeEghJ'), 'AAEuAAAAAABmM9ooHmGkSqQvHNKEhEhoAQCHacGtJwrVS7ruKbQ/uu5BAADfzalgAAA=': Messages(PublicFoldersRoot(<exchangelib.account.Account object at 0x105233290>, '[self]', 'IPM_SUBTREE', 0, 0, 5, None, 'AQEuAAADGkRzkKpmEc2byACqAC/EWgMAh2nBrScK1Uu67im0P7ruQQAAAwIAAAA=', 'AQAAABYAAACHacGtJwrVS7ruKbQ/uu5BAADeEghM'), 'test', 0, 0, 0, 'IPF.Note', 'AAEuAAAAAABmM9ooHmGkSqQvHNKEhEhoAQCHacGtJwrVS7ruKbQ/uu5BAADfzalgAAA=', 'AQAAABYAAACHacGtJwrVS7ruKbQ/uu5BAADf/yUc')}

And for a first level folder in public_folders_root

Messages(PublicFoldersRoot(<exchangelib.account.Account object at 0x105233290>, '[self]', 'IPM_SUBTREE', 0, 0, 5, None, 'AQEuAAADGkRzkKpmEc2byACqAC/EWgMAh2nBrScK1Uu67im0P7ruQQAAAwIAAAA=', 'AQAAABYAAACHacGtJwrVS7ruKbQ/uu5BAADeEghM'), 'test', 0, 0, 0, 'IPF.Note', 'AAEuAAAAAABmM9ooHmGkSqQvHNKEhEhoAQCHacGtJwrVS7ruKbQ/uu5BAADfzalgAAA=', 'AQAAABYAAACHacGtJwrVS7ruKbQ/uu5BAADf/yUc')

the according parent_folder_id is set to

ParentFolderId(id='AQEuAAADZjPaKB5hpEqkLxzShIQASGgBAIdpwa0nCtVLuu4ptD+67kEAAAMCAAAA', changekey='AQAAAA==')

There is clearly a mismatch..

[ecederstrand]

Ah, maybe what happens is that parent_folder_id for the public folder points to the root folder of the account owning the folder, not the public_folders_root of the current account. That would explain things.

You could try a patch like this:

--- a/exchangelib/folders/roots.py
+++ b/exchangelib/folders/roots.py
@@ -332,6 +332,17 @@ class PublicFoldersRoot(RootOfHierarchy):
         if self._distinguished_id:
             self._distinguished_id.mailbox = None  # See DistinguishedFolderId.clean()
 
+    @property
+    def _folders_map(self):
+        # Top-level public folders may point to the root folder of the owning account and not the public folders root
+        # of this account. This breaks the assumption of get_children(). Fix it by overwriting the parent folder.
+        fix_parents = self._subfolders is None
+        res = super()._folders_map
+        if fix_parents:
+            with self._subfolders_lock:
+                for f in res.values():
+                    if f.id != self.id:
+                        f.parent = self
+        return res
+
     def get_children(self, folder):
         # EWS does not allow deep traversal of public folders, so self._folders_map will only populate the top-level
         # subfolders. To traverse public folders at arbitrary depth, we need to get child folders on demand.

[ecederstrand]

Available as #1316

[iglimanaj]

Reply to #1300 (comment)

@ecederstrand first of all these two changes are necessary. (I could not push in your branch)

Then the following error occurred:

Exception has occurred: AttributeError
property 'parent' of 'PublicFoldersRoot' object has no setter
KeyError: 'folders'

During handling of the above exception, another exception occurred:

  File "/Users/iglimanaj/Desktop/ambersearch/exchangelib/exchangelib/properties.py", line 287, in __setattr__
    return super().__setattr__(key, value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iglimanaj/Desktop/ambersearch/exchangelib/exchangelib/folders/roots.py", line 344, in _folders_map
    f.parent = None if f.id == self.id else self
    ^^^^^^^^
  File "/Users/iglimanaj/Desktop/ambersearch/exchangelib/exchangelib/folders/roots.py", line 98, in get_children
    for f in self._folders_map.values():
             ^^^^^^^^^^^^^^^^^
  File "/Users/iglimanaj/Desktop/ambersearch/exchangelib/exchangelib/folders/roots.py", line 352, in get_children
    children = list(super().get_children(folder=folder))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iglimanaj/Desktop/ambersearch/exchangelib/exchangelib/folders/collections.py", line 43, in folders
    return tuple(self._folders)
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/iglimanaj/Desktop/ambersearch/exchangelib/exchangelib/folders/collections.py", line 46, in __len__
    return len(self.folders)
               ^^^^^^^^^^^^
  File "/Users/iglimanaj/Desktop/ambersearch/exchangelib/igli_test.py", line 27, in <module>
    children = list(public_folders_root.children)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: property 'parent' of 'PublicFoldersRoot' object has no setter

[ecederstrand]

Sorry about that. Fixed now.

[iglimanaj]

Alright, thx! The problem stated above is present anyways AttributeError: property 'parent' of 'PublicFoldersRoot' object has no setter

[ecederstrand]

Fixed

[iglimanaj]

I now can retrieve the folders. Great. I will test in another environment to double check and let you know. Huge thanks! @ecederstrand

[iglimanaj]

I tested in different environments, as promised. The fix in the pull request looks everywhere fine!

@ecederstrand is it possible to make a new tag/release, once the pull request is merged?