faster rcnn train resnet101 problem
assign pretrain model gamma to bn4b5_branch2c
2017-12-04 18:57:50.782414: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [1,35,63,36] vs. [300,116]
[[Node: gradients/sub_1_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"](gradients/sub_1_grad/Shape, gradients/sub_1_grad/Shape_1/_1069)]]
2017-12-04 18:57:50.782494: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [1,35,63,36] vs. [300,116]
[[Node: gradients/sub_1_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"](gradients/sub_1_grad/Shape, gradients/sub_1_grad/Shape_1/_1069)]]
2017-12-04 18:57:50.782522: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [1,35,63,36] vs. [300,116]
[[Node: gradients/sub_1_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"](gradients/sub_1_grad/Shape, gradients/sub_1_grad/Shape_1/_1069)]]
2017-12-04 18:57:50.782546: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [1,35,63,36] vs. [300,116]
[[Node: gradients/sub_1_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"](gradients/sub_1_grad/Shape, gradients/sub_1_grad/Shape_1/_1069)]]
2017-12-04 18:57:50.800155: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [1,35,63,36] vs. [300,116]
[[Node: gradients/sub_1_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"](gradients/sub_1_grad/Shape, gradients/sub_1_grad/Shape_1/_1069)]]
Traceback (most recent call last):
File "./faster_rcnn/train_net.py", line 109, in <module>
restore=bool(float(args.restore)))
File "./faster_rcnn/../lib/fast_rcnn/train.py", line 403, in train_net
sw.train_model(sess, max_iters, restore=restore)
File "./faster_rcnn/../lib/fast_rcnn/train.py", line 258, in train_model
cls_prob, bbox_pred, rois = sess.run(fetches=fetch_list, feed_dict=feed_dict)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 895, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1124, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1321, in _do_run
options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1340, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [1,35,63,36] vs. [300,116]
[[Node: gradients/sub_1_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"](gradients/sub_1_grad/Shape, gradients/sub_1_grad/Shape_1/_1069)]]
[[Node: Mean_2/_1101 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_2700_Mean_2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
Caused by op u'gradients/sub_1_grad/BroadcastGradientArgs', defined at:
File "./faster_rcnn/train_net.py", line 109, in <module>
restore=bool(float(args.restore)))
File "./faster_rcnn/../lib/fast_rcnn/train.py", line 403, in train_net
sw.train_model(sess, max_iters, restore=restore)
File "./faster_rcnn/../lib/fast_rcnn/train.py", line 143, in train_model
grads, norm = tf.clip_by_global_norm(tf.gradients(loss, tvars), 20)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gradients_impl.py", line 542, in gradients
grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gradients_impl.py", line 348, in _MaybeCompile
return grad_fn() # Exit early
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gradients_impl.py", line 542, in <lambda>
grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_grad.py", line 700, in _SubGrad
rx, ry = gen_array_ops._broadcast_gradient_args(sx, sy)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 395, in _broadcast_gradient_args
name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2628, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1204, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
...which was originally created as op u'sub_1', defined at:
File "./faster_rcnn/train_net.py", line 109, in <module>
restore=bool(float(args.restore)))
[elided 0 identical lines from previous traceback]
File "./faster_rcnn/../lib/fast_rcnn/train.py", line 403, in train_net
sw.train_model(sess, max_iters, restore=restore)
File "./faster_rcnn/../lib/fast_rcnn/train.py", line 112, in train_model
self.net.build_loss(ohem=cfg.TRAIN.OHEM)
File "./faster_rcnn/../lib/networks/network.py", line 667, in build_loss
bbox_outside_weights * self.smooth_l1_dist(bbox_inside_weights * (bbox_pred - bbox_targets)), \
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py", line 865, in binary_op_wrapper
return func(x, y, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_math_ops.py", line 2631, in _sub
result = _op_def_lib.apply_op("Sub", x=x, y=y, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2628, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1204, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
InvalidArgumentError (see above for traceback): Incompatible shapes: [1,35,63,36] vs. [300,116]
[[Node: gradients/sub_1_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"](gradients/sub_1_grad/Shape, gradients/sub_1_grad/Shape_1/_1069)]]
[[Node: Mean_2/_1101 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_2700_Mean_2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
怎么解决????谁知道,thanks!!!!