TensorFlow Object Detection目标检测模型训练时Loss急剧上升直至为NAN

今天用TensorFlow Object Detection训练目标检测模型的时候,Loss一直不正常,先下降了一下,然后急剧上升直至为NAN。然后报错Model diverged with loss = NaN.。截取部分日志如下:

INFO:tensorflow:loss = 18919772.0, step = 0
INFO:tensorflow:loss = 344412.66, step = 100 (27.579 sec)
INFO:tensorflow:loss = 156323.77, step = 200 (22.843 sec)
INFO:tensorflow:loss = 286260.12, step = 300 (22.834 sec)
INFO:tensorflow:loss = 7225620.0, step = 400 (22.840 sec)
INFO:tensorflow:loss = 35882144.0, step = 500 (22.831 sec)
INFO:tensorflow:loss = 11317121000.0, step = 600 (22.844 sec)
INFO:tensorflow:loss = 264382550000.0, step = 700 (22.859 sec)
INFO:tensorflow:loss = 2169563800000.0, step = 800 (22.870 sec)
INFO:tensorflow:loss = 49792570000000.0, step = 900 (22.838 sec)
INFO:tensorflow:loss = 279824520000000.0, step = 1000 (22.857 sec)
INFO:tensorflow:loss = 610852500000000.0, step = 1100 (22.872 sec)
INFO:tensorflow:loss = 8140467300000000.0, step = 1200 (22.867 sec)
INFO:tensorflow:loss = 1.5560248e+16, step = 1300 (22.864 sec)
ERROR:tensorflow:Model diverged with loss = NaN.

这情况一看就感觉像是个低级错误引起的,随手一查果然立刻恍然发现,config文件的num_classes的值忘记了修改
网上还看到有人说造成这种现象的原因还可能有:
label_map.pbtxt中的name和生成的tfrecord的类别名称不一致的
特此记录,以防万一。

已标记关键词 清除标记
大家好, 新手刚刚学,IDE:spyder 预测一个停车场车辆的驶出率, 输入(间和车辆驶入率)二维。 训练的数据就是,驶出率car/min,间(间:1代表一天,从凌晨10秒=10/24/3600的候开始到晚上23点多),驶入率car/min, 只建了一层的hidden layer,然后print loss是都是nan... 不知道哪里出了问题,是因为层太简单了么?还是激活函数有问题呢? 看网上说排除零的影响,我把输入数和输出数都+1,变得非零了也还是nan... 代码如下: ``` import tensorflow as tf import numpy as np import pandas as pd data=pd.read_csv('0831new.csv') date=data['date'] erate=data['erate'] x=pd.concat([date,erate],axis=1) drate=data['drate'] y=np.array(drate) x=np.array(x) y=y.reshape([7112,1]) x=x+1 y=y+1 z=[] def add_layer(inputs, in_size, out_size, activation_function=None): # add one more layer and return the output of this layer Weights = tf.Variable(tf.random_normal([in_size, out_size])) biases = tf.Variable(tf.zeros([1, out_size]) + 0.01) Wx_plus_b = tf.matmul(inputs, Weights) + biases if activation_function is None: outputs = Wx_plus_b else: outputs = activation_function(Wx_plus_b) return outputs xs = tf.placeholder(tf.float32, [None, 2]) ys = tf.placeholder(tf.float32, [None, 1]) l1 = add_layer(xs, 2, 5, activation_function=tf.nn.tanh) prediction = add_layer(l1,5, 1, activation_function=None) loss = tf.reduce_mean(tf.reduce_sum(tf.square(ys - prediction), reduction_indices=[1,0])) train_step= tf.train.GradientDescentOptimizer(0.001).minimize(loss) if int((tf.__version__).split('.')[1]) < 12: init = tf.initialize_all_variables() else: init = tf.global_variables_initializer() sess = tf.Session() sess.run(init) for i in range(1000): # training sess.run(train_step, feed_dict={xs: x, ys: y}) if i % 25 == 0: # to see the step improvement print('loss:',sess.run(loss, feed_dict={xs:x, ys:y})) z.append(loss) ``` ![图片说明](https://img-ask.csdn.net/upload/201909/20/1568984044_999083.png) 帮忙给件建议吧~ 谢谢
©️2020 CSDN 皮肤主题: 编程工作室 设计师:CSDN官方博客 返回首页