Why Is My Loss Function Return Negative Values?

I am studying to Recommendation System. I use to Random Forest with Tensorflow. I have a problem with my loss result. How to fix my code. Help me.

This is value of x_data
shape=(6000,116)
value is 0 or 1

array([[1, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 1, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       ...,
       [0, 0, 0, ..., 1, 1, 0],
       [0, 0, 0, ..., 0, 0, 1],
       [0, 0, 0, ..., 0, 0, 1]])

This is value of y_data
shape=(6000,1)
value is 0 or 1

array([[0],
       [0],
       [1],
       ...,
       [0],
       [0],
       [0]])

This is my code

def next_batch(x_data, y_data, batch_size):
    if (len(x_data) != len(y_data)):
        return None, None

    batch_mask = np.random.choice(len(x_data), batch_size)
    x_batch = x_data[batch_mask]
    y_batch = y_data[batch_mask]
    return x_batch, y_batch

x_train = train.iloc[:, 3:].values
y_train = train.iloc[:,2:3].values
x_test = test.iloc[:,2:].values

x_data = np.array(x_train, dtype=np.float32)
y_data = np.array(y_train, dtype=np.int64)
test_data = np.array(x_test, dtype=np.float32)

# Parameters
num_steps = 500 
batch_size = 1024
num_classes = 2 
num_features = 116
num_trees = 10
max_nodes = 1000

tf.reset_default_graph()

# Input and Target placeholders
X = tf.placeholder(tf.float32, shape=[None, num_features])
Y = tf.placeholder(tf.int64, shape=[None,1])

# Random Forest Parameters
hparams = tensor_forest.ForestHParams(num_classes=num_classes,
                                      num_features=num_features,
                                      num_trees=num_trees,
                                      max_nodes=max_nodes).fill()


#Build the Random Forest
forest_graph = tensor_forest.RandomForestGraphs(hparams)

# Get training graph and loss
train_op = forest_graph.training_graph(X, Y)
loss_op = forest_graph.training_loss(X,Y)

# Measure the accuracy
infer_op, _, _ = forest_graph.inference_graph(X)
correct_prediction = tf.equal(tf.argmax(infer_op, 1), tf.cast(Y, tf.int64))
accuracy_op = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

init_vars = tf.group(tf.global_variables_initializer(), resources.initialize_resources(resources.shared_resources()))

sess = tf.Session()
sess.run(init_vars)

# Training
for i in range(1, num_steps + 1):
    # Prepare Data
    # Get the next batch of MNIST data (only images are needed, not labels)
    batch_x, batch_y = next_batch(x_data, y_data, batch_size)
    _, l = sess.run([train_op, loss_op], feed_dict={X: batch_x, Y: batch_y})
    if i % 50 == 0 or i == 1:
        acc = sess.run(accuracy_op, feed_dict={X: batch_x, Y: batch_y})
        print('Step %i, Loss: %f, Acc: %f' % (i, l, acc))

why is my loss function return negative values?
Result

INFO:tensorflow:Constructing forest with params = 
INFO:tensorflow:{'num_trees': 10, 'max_nodes': 1000, 'bagging_fraction': 1.0, 'feature_bagging_fraction': 1.0, 'num_splits_to_consider': 10, 'max_fertile_nodes': 0, 'split_after_samples': 250, 'valid_leaf_threshold': 1, 'dominate_method': 'bootstrap', 'dominate_fraction': 0.99, 'model_name': 'all_dense', 'split_finish_name': 'basic', 'split_pruning_name': 'none', 'collate_examples': False, 'checkpoint_stats': False, 'use_running_stats_method': False, 'initialize_average_splits': False, 'inference_tree_paths': False, 'param_file': None, 'split_name': 'less_or_equal', 'early_finish_check_every_samples': 0, 'prune_every_samples': 0, 'num_classes': 2, 'num_features': 116, 'bagged_num_features': 116, 'bagged_features': None, 'regression': False, 'num_outputs': 1, 'num_output_columns': 3, 'base_random_seed': 0, 'leaf_model_type': 0, 'stats_model_type': 0, 'finish_type': 0, 'pruning_type': 0, 'split_type': 0}
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/contrib/tensor_forest/python/tensor_forest.py:529: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
Step 1, Loss: -1.000000, Acc: 0.873047
Step 50, Loss: -250.399994, Acc: 0.833313
Step 100, Loss: -537.200012, Acc: 0.856388
Step 150, Loss: -822.799988, Acc: 0.841568
Step 200, Loss: -1001.000000, Acc: 0.835522
Step 250, Loss: -1001.000000, Acc: 0.839737
Step 300, Loss: -1001.000000, Acc: 0.817566
Step 350, Loss: -1001.000000, Acc: 0.816372
Step 400, Loss: -1001.000000, Acc: 0.843414
Step 450, Loss: -1001.000000, Acc: 0.829651
Step 500, Loss: -1001.000000, Acc: 0.839970

1 Answer

The loss is just a scalar that you are trying to minimize. It's not supposed to be positive.

One of the reason you are getting negative values in loss is because the training_loss in RandomForestGraphs is implemented using cross entropy loss or negative log liklihood as per the reference code here.

Also, as you can see the loss remains constant in the later iterations, I suppose doing Hyperparameter Tuning will make the tree robust to variances of the data. You can reference some ideas from here.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Chloe Bennett

Chloe Bennett

Culture, Media & Entertainment Columnist

Chloe Bennett explores the intersection of pop culture, streaming entertainment, digital trends, and contemporary lifestyle. Her weekly commentary reaches thousands of culture enthusiasts.

Share this article
Twitter Facebook Pinterest