[Question] Why 'broadcast' returns value that not equals to param in any rank?

[Nov11]

Here is what I tried to do:

1. Create random value in each rank. Call it x.
2. Broadcast from root rank0.Call it y.
3. Use the value to init a variable. Call it v.

I suppose broad cast returns random value generated on rank0. And variable v should be same as that value.
So given a run of two process, y(rank0) == y(rank1) == x(rank0), and v(rank0) == v(rank1) == x(rank0).

But outputs of the code below shows different result.

y(rank0) == y(rank1) but y(rank0) != x(rank0)
v(rank0) == v(rank1) but v(rank0) != x(rank0), v(rank0) != y(rank0)

why would this happen?

import horovod.tensorflow as hvd
import numpy as np

hvd.init()

# import tensorflow as tf
import tensorflow._api.v2.compat.v1 as tf

tf.disable_eager_execution()

gpus = tf.config.experimental.list_physical_devices('GPU')
for gpu in gpus:
    tf.config.experimental.set_memory_growth(gpu, True)
if gpus:
    tf.config.experimental.set_visible_devices(gpus[hvd.local_rank()], 'GPU')


if __name__ == '__main__':
    x = tf.random_normal_initializer(seed=hvd.local_rank())(shape=[5,3]) # x : random init value
    print('x', x)
    y = hvd.broadcast(x, root_rank=0) # y : recevie init value from root rank
    print('y', y)

    v = tf.Variable(y)  # init local variable from y
    print('v', v)
    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        print('x : ', sess.run(x)) # show local random init value
        print('y : ', sess.run(y)) # should be same as x on rank0
        print('v : ', sess.run(v)) # should be same as y
    print('exit')

outputs:

[1] 2023-04-18 12:01:06.322022: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
[1] To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
[0] 2023-04-18 12:01:06.325591: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
[0] To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
[0] 2023-04-18 12:01:07.495962: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 30939 MB memory:  -> device: 0, name: Tesla V100-SXM2-32GB, pci bus id: 0000:1a:00.0, compute capability: 7.0
[1] 2023-04-18 12:01:08.501129: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 30939 MB memory:  -> device: 1, name: Tesla V100-SXM2-32GB, pci bus id: 0000:1b:00.0, compute capability: 7.0
[1] x Tensor("random_normal:0", shape=(5, 3), dtype=float32)
[1] y Tensor("HorovodBroadcast_random_normal_0:0", shape=(5, 3), dtype=float32)
[1] v <tf.Variable 'Variable:0' shape=(5, 3) dtype=float32>
[1] x :  [[ 1.1944131   0.15362588 -0.73293823]
[1]  [ 1.313663    1.190688    1.7747383 ]
[1]  [-0.9063124  -0.57891583 -1.4952853 ]
[1]  [ 0.14176413  1.2206951   0.5467055 ]
[1]  [ 0.27716216  1.1724057   0.07568353]]
[1] y :  [[ 1.7186217   0.16241188 -0.5876202 ]
[1]  [-1.067236   -0.5135878   1.3255594 ]
[1]  [ 0.28959653 -0.8174958   0.9435837 ]
[1]  [ 1.8280293   2.139199    0.844802  ]
[1]  [ 0.2921385  -0.54440576 -0.7108431 ]]
[1] v :  [[-0.39915758  2.1044393   0.17107224]
[1]  [ 0.54651815 -2.4234028   0.422554  ]
[1]  [ 0.2894379  -0.5043041  -0.96068907]
[1]  [-0.65109813  0.11453361 -0.1035478 ]
[1]  [-1.300912   -0.50387573  0.02013604]]
[1] exit
[0] x Tensor("random_normal:0", shape=(5, 3), dtype=float32)
[0] y Tensor("HorovodBroadcast_random_normal_0:0", shape=(5, 3), dtype=float32)
[0] v <tf.Variable 'Variable:0' shape=(5, 3) dtype=float32>
[0] x :  [[-2.3260818  -0.7108519  -0.27827743]
[0]  [-1.654401   -1.9124602   0.42695242]
[0]  [ 0.10304912 -0.93675536  2.0990272 ]
[0]  [ 1.4418154   0.57435626  1.7707795 ]
[0]  [-0.6538431  -0.22350888 -0.61087763]]
[0] y :  [[ 1.7186217   0.16241188 -0.5876202 ]
[0]  [-1.067236   -0.5135878   1.3255594 ]
[0]  [ 0.28959653 -0.8174958   0.9435837 ]
[0]  [ 1.8280293   2.139199    0.844802  ]
[0]  [ 0.2921385  -0.54440576 -0.7108431 ]]
[0] v :  [[-0.39915758  2.1044393   0.17107224]
[0]  [ 0.54651815 -2.4234028   0.422554  ]
[0]  [ 0.2894379  -0.5043041  -0.96068907]
[0]  [-0.65109813  0.11453361 -0.1035478 ]
[0]  [-1.300912   -0.50387573  0.02013604]]
[0] exit

[maxhgerlach]

        print('x : ', sess.run(x)) # show local random init value
        print('y : ', sess.run(y)) # should be same as x on rank0
        print('v : ', sess.run(v)) # should be same as y

Merge these to the same session run() or use eager mode. Else y won't persist.