DevicePool #8615

aliuTT · 2024-05-17T19:08:14Z

PR for DevicePool. Merging a few concepts in the codebase that is more or less managing multi devices across runtime:

ActiveDevices: tracking some FW builds, device init
tt_metal/DeviceMesh: opens a bunch of devices and returns handles to needed devices. DevicePool will basically be a rename of this.
ttnn::device::device_pool::devices: global handle for all devices, this will be deleted

I'm also trying out something that isn't really tested in backend:

Device *dev = new Device();
// run some tests
dev->close();
dev->initialize();
// run some tests

I'll also clean up the instance method to be something like this:

tt::DevicePool::initialize(device_ids, num_hw_cqs, l1_small_size);
tt::DevicePool::instance().get_all_devices();

aliuTT · 2024-05-29T20:44:47Z

tt_metal/impl/device/device_pool.hpp

+      std::cout << " DP initialize " << std::endl;
+      std::cout << " device ids ";
+      for (const auto &chip: device_ids) { std::cout << chip << " " << std::endl; }
+      std::cout << " num hw cqs " << num_hw_cqs << " l1 small " << l1_small_size << std::endl;


will remove prints

thanks, pls re-ping for approval once done

aliuTT · 2024-05-29T20:44:58Z

tt_metal/impl/device/device_pool.cpp

+
+    } else {
+      const auto& dev = this->devices[id];
+      std::cout << " DP re-init device " << id << std::endl;


will remove prints

xanderchin · 2024-05-29T23:05:08Z

tests/ttnn/unit_tests/operations/test_conv2d.py

@@ -373,6 +373,7 @@ def test_resnet50_conv_gs(
    pad_w,
    use_1d_systolic_array,
 ):
+    pytest.skip("Skipping, suspecting some device state dependency")


Is there an issue for this being disabled?

make sure these issues are resolved before merge

xanderchin · 2024-05-29T23:06:08Z

tt_metal/impl/device/device.cpp

@@ -179,6 +136,7 @@ void Device::initialize_build() {
 }

 void Device::build_firmware() {
+  std::cout << " BUILDING FIRMWARE " << this->id() << std::endl;


abhullar-tt · 2024-05-29T20:56:08Z

tt_metal/detail/tt_metal.hpp

@@ -39,13 +41,13 @@ namespace tt::tt_metal{
        }

        std::map<chip_id_t, Device *> CreateDevices(
+            // TODO: delete this in favour of DevicePool


will this be deleted as part of a sep PR?

abhullar-tt · 2024-05-29T20:56:23Z

tt_metal/detail/tt_metal.hpp

@@ -155,6 +157,8 @@ namespace tt::tt_metal{
         * | device        | The device holding the program being profiled.    | Device *        |                           | True     |
         * */
 	    void InitDeviceProfiler(Device *device);
+      // API to clear all device profilers
+	    void ClearDeviceProfiler();


ClearAllDeviceProfilers?

abhullar-tt · 2024-05-29T20:57:58Z

tt_metal/impl/device/device.cpp

@@ -1325,7 +1261,8 @@ bool Device::close() {
                        not_done_dispatch_cores.insert(phys_core);
                        log_debug(tt::LogMetal, "MMIO Device Prefetch core: Logical: {} - Physical: {}", prefetch_location.str(), phys_core.str());
                    }
-                } else if (this->active_devices_.is_device_active(device_id)) {
+                } else  {
+                    // Assume that R chip is always open


should we assert for this?

aliuTT had a problem deploying to dev May 19, 2024 01:46 — with GitHub Actions Error

aliuTT had a problem deploying to dev May 19, 2024 01:46 — with GitHub Actions Failure

aliuTT had a problem deploying to dev May 21, 2024 14:26 — with GitHub Actions Failure