faster rcnn train resnet101 problem

liningxiao2017 2017-12-04 07:00:38
assign pretrain model gamma to bn4b5_branch2c
2017-12-04 18:57:50.782414: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [1,35,63,36] vs. [300,116]
[[Node: gradients/sub_1_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"](gradients/sub_1_grad/Shape, gradients/sub_1_grad/Shape_1/_1069)]]
2017-12-04 18:57:50.782494: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [1,35,63,36] vs. [300,116]
[[Node: gradients/sub_1_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"](gradients/sub_1_grad/Shape, gradients/sub_1_grad/Shape_1/_1069)]]
2017-12-04 18:57:50.782522: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [1,35,63,36] vs. [300,116]
[[Node: gradients/sub_1_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"](gradients/sub_1_grad/Shape, gradients/sub_1_grad/Shape_1/_1069)]]
2017-12-04 18:57:50.782546: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [1,35,63,36] vs. [300,116]
[[Node: gradients/sub_1_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"](gradients/sub_1_grad/Shape, gradients/sub_1_grad/Shape_1/_1069)]]
2017-12-04 18:57:50.800155: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [1,35,63,36] vs. [300,116]
[[Node: gradients/sub_1_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"](gradients/sub_1_grad/Shape, gradients/sub_1_grad/Shape_1/_1069)]]
Traceback (most recent call last):
File "./faster_rcnn/train_net.py", line 109, in <module>
restore=bool(float(args.restore)))
File "./faster_rcnn/../lib/fast_rcnn/train.py", line 403, in train_net
sw.train_model(sess, max_iters, restore=restore)
File "./faster_rcnn/../lib/fast_rcnn/train.py", line 258, in train_model
cls_prob, bbox_pred, rois = sess.run(fetches=fetch_list, feed_dict=feed_dict)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 895, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1124, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1321, in _do_run
options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1340, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [1,35,63,36] vs. [300,116]
[[Node: gradients/sub_1_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"](gradients/sub_1_grad/Shape, gradients/sub_1_grad/Shape_1/_1069)]]
[[Node: Mean_2/_1101 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_2700_Mean_2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Caused by op u'gradients/sub_1_grad/BroadcastGradientArgs', defined at:
File "./faster_rcnn/train_net.py", line 109, in <module>
restore=bool(float(args.restore)))
File "./faster_rcnn/../lib/fast_rcnn/train.py", line 403, in train_net
sw.train_model(sess, max_iters, restore=restore)
File "./faster_rcnn/../lib/fast_rcnn/train.py", line 143, in train_model
grads, norm = tf.clip_by_global_norm(tf.gradients(loss, tvars), 20)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gradients_impl.py", line 542, in gradients
grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gradients_impl.py", line 348, in _MaybeCompile
return grad_fn() # Exit early
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gradients_impl.py", line 542, in <lambda>
grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_grad.py", line 700, in _SubGrad
rx, ry = gen_array_ops._broadcast_gradient_args(sx, sy)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 395, in _broadcast_gradient_args
name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2628, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1204, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

...which was originally created as op u'sub_1', defined at:
File "./faster_rcnn/train_net.py", line 109, in <module>
restore=bool(float(args.restore)))
[elided 0 identical lines from previous traceback]
File "./faster_rcnn/../lib/fast_rcnn/train.py", line 403, in train_net
sw.train_model(sess, max_iters, restore=restore)
File "./faster_rcnn/../lib/fast_rcnn/train.py", line 112, in train_model
self.net.build_loss(ohem=cfg.TRAIN.OHEM)
File "./faster_rcnn/../lib/networks/network.py", line 667, in build_loss
bbox_outside_weights * self.smooth_l1_dist(bbox_inside_weights * (bbox_pred - bbox_targets)), \
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py", line 865, in binary_op_wrapper
return func(x, y, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_math_ops.py", line 2631, in _sub
result = _op_def_lib.apply_op("Sub", x=x, y=y, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2628, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1204, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Incompatible shapes: [1,35,63,36] vs. [300,116]

[[Node: gradients/sub_1_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"](gradients/sub_1_grad/Shape, gradients/sub_1_grad/Shape_1/_1069)]]
[[Node: Mean_2/_1101 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_2700_Mean_2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

怎么解决????谁知道,thanks!!!!
...全文
945 2 打赏 收藏 转发到动态 举报
写回复
用AI写文章
2 条回复
切换为时间正序
请发表友善的回复…
发表回复
liningxiao2017 2018-03-11
  • 打赏
  • 举报
回复
我的是因为box_loss计算时tensor不一致
HAN_ALPHA 2018-01-14
  • 打赏
  • 举报
回复
你好,这个问题解决了吗?我也遇到了一样的问题

37,718

社区成员

发帖
与我相关
我的任务
社区描述
JavaScript,VBScript,AngleScript,ActionScript,Shell,Perl,Ruby,Lua,Tcl,Scala,MaxScript 等脚本语言交流。
社区管理员
  • 脚本语言(Perl/Python)社区
  • IT.BOB
加入社区
  • 近7日
  • 近30日
  • 至今

试试用AI创作助手写篇文章吧