Training Deep Neural Networks: Keras, TensorFlow, and Tabular Data

Training a neural network means turning a mathematical prediction rule into an experiment: prepare the data, choose an objective, update parameters, and measure performance on unseen examples. This lesson connects the training loop to Keras and TensorFlow, then develops a tabular classification workflow with numerical and categorical features.

Learning goals

After this lesson you should be able to choose an output and loss for regression or classification, distinguish batches from epochs, build a Functional API model, prepare tabular data without leakage, use early stopping, and interpret performance against a baseline.

1. What is given, and what do you choose?

The problem defines the target’s meaning and the information available at prediction time. A target such as “disease present at examination” is different from “new disease within the next year.” The latter requires longitudinal follow-up, a clear starting point, and appropriate handling of observation windows.

You choose which available measurements to use, how to encode them, and how to build and train the model.

Choice	Example	Why it matters
Hidden layers and widths	One hidden layer with 16 units	Controls capacity and computation
Hidden activation	ReLU	Allows nonlinear relationships
Output representation	One sigmoid probability	Matches binary classification
Loss	Binary cross-entropy	Defines the training objective
Optimizer and learning rate	Adam with learning rate 0.001	Controls parameter updates
Batch size	32 observations	Controls examples per gradient estimate
Training duration	Up to 200 epochs with early stopping	Balances fitting and generalization

Weights and biases are learned parameters. Architecture, learning rate, and batch size are hyperparameters selected using training and validation evidence. The test set must not become another tuning set.

2. The training loop, precisely

For a batch of inputs X and labels y:

1. Forward pass: compute hidden activations and predictions from the current parameters.
2. Loss calculation: reduce prediction errors to a scalar batch objective.
3. Backpropagation: apply automatic differentiation and the chain rule to calculate gradients of that objective with respect to parameters.
4. Optimizer update: use the gradients and any optimizer state to update weights and biases.

Backpropagation computes derivatives; the optimizer decides the update. In a TensorFlow custom loop, a gradient tape can calculate derivatives and an optimizer can apply them. Keras fit() coordinates these operations for you. GPU acceleration can speed up tensor operations, but is not required for backpropagation or this small example.

Plain gradient descent uses θ ← θ − α∇θL, where θ contains the parameters and α is the learning rate. This is a local downhill direction for sufficiently small steps, not a guarantee of a global optimum.

Worked update. If a weight is 0.40, its batch gradient is 0.25, and α=0.02, its new value is 0.40−0.02×0.25=0.395. Adam generally produces a different step because it maintains running estimates of gradients and squared gradients.

3. Choose a loss that matches the output

Continuous targets: mean squared error

For B scalar predictions, MSE=(1/B)Σᵢ(yᵢ−ŷᵢ)². A linear output can represent any real number; positive-only targets may call for a transformation or a constrained output.

For targets [2,4] and predictions [3,2], MSE is (1²+(−2)²)/2=2.5. Its derivative with respect to prediction i is 2(ŷᵢ−yᵢ)/B. Large residuals receive large squared penalties, making MSE sensitive to outliers. MAE or Huber loss may be useful alternatives depending on the task.

Binary targets: binary cross-entropy

For y∈{0,1} and predicted probability p:

ℓ(y,p)=−y log(p)−(1−y)log(1−p).

For y=1, predicting p=0.9 gives about 0.1054 loss; p=0.01 gives about 4.6052. A confident wrong prediction is penalized strongly. Logs are natural logarithms.

Two consistent Keras designs are:

• Dense(1, activation="sigmoid") with BinaryCrossentropy(from_logits=False).
• Dense(1) with BinaryCrossentropy(from_logits=True).

The second returns raw scores called logits and lets the loss handle the probability transformation stably. Do not combine a sigmoid output with from_logits=True. This lesson uses the first design for an immediately interpretable probability output.

Loss and metrics have different roles. Cross-entropy supplies differentiable training feedback. Accuracy measures thresholded decisions and is reported for interpretation. A lower loss need not improve accuracy if probabilities change without crossing the threshold.

4. Mini-batches, SGD, Adam, and epochs

The full-data objective is L(θ)=(1/n)Σᵢℓᵢ(θ). Full-batch gradient descent evaluates all n examples per update. That is feasible for some datasets, but can be expensive at scale. A uniformly sampled mini-batch estimates its gradient at lower per-step cost. Under suitable sampling and ordinary per-example losses, that estimate is unbiased; this does not guarantee unbiased final parameters.

Strictly, stochastic gradient descent can use one example per update; the term SGD is also commonly used for mini-batch updates. For ordinary independent tabular rows, reshuffling each epoch is useful. Time-dependent and grouped data require additional care.

Stochasticity can help optimization explore a nonconvex landscape, but does not guarantee escape from every poor minimum or saddle point. Adam is a widely used adaptive optimizer, not a universally best default. Compare validation behavior and tune the learning rate rather than treating an optimizer name as a solution.

An epoch is a pass through the training set. A batch is one group of examples, and an iteration is one update. For 242 training rows and batch size 32, retaining the incomplete batch gives ceil(242/32)=8 updates per epoch. Ten epochs give 80 updates. Early stopping can reduce that total.

Changing batch size changes the number of updates per epoch as well as gradient noise. Compare experiments using more than epoch counts alone. For a small table, GPU setup and transfer overhead may outweigh any acceleration.

5. Keras and TensorFlow: who does what?

TensorFlow supplies tensor operations, automatic differentiation, execution, and device support. Keras provides model-building and training abstractions. Modern Keras 3 is multi-backend: it can train using TensorFlow, JAX, or PyTorch. Here we explicitly choose TensorFlow before importing Keras. See the Keras setup guide for backend and installation details.

The three model-building styles serve different needs:

API	Structure	Suitable use
Sequential	A simple stack of layers	One input, one output, no branching
Functional	A graph connecting symbolic inputs and layers	Multiple inputs, shared layers, branching, or reusable components
Subclassing	A Python class defining call()	Custom model behavior requiring more explicit control

We use the Functional API because the graph is explicit and easy to extend.

import os
os.environ["KERAS_BACKEND"] = "tensorflow"  # Before importing keras.
import keras

inputs = keras.Input(shape=(29,), name="features")
h = keras.layers.Dense(16, activation="relu")(inputs)
outputs = keras.layers.Dense(1, activation="sigmoid")(h)
model = keras.Model(inputs=inputs, outputs=outputs)

The comma in (29,) creates a one-element tuple. The shape describes one example, excluding the batch dimension. A batch of 32 examples has shape (32,29); the symbolic input typically has shape (None,29).

The first dense layer has 29×16+16=480 parameters, and the output layer has 16×1+1=17. Total: 497. To create a second hidden layer, apply another Dense layer to h before the output. Keep capacity justified by evidence rather than adding depth automatically.

6. Heart-disease data: define the experiment honestly

The official Keras structured-data example uses a Cleveland-derived CSV with 303 rows, 13 input features, and a binary disease-presence label. The UCI Heart Disease documentation describes the original diagnosis target and missing values; versions and preprocessing can differ. Do not assume every file called “heart disease” has identical labels or records.

The CSV used below encodes its binary outcome in target. Its target is existing disease presence, not verified one-year incidence. An original UCI severity target with values 0–4 would need an explicitly justified conversion, commonly 0 versus values greater than 0. Do not apply that conversion blindly to a file already encoded as binary.

The supplied course outline reports 72.6% accuracy for predicting all zeros and about 83.6% for a neural network. Those figures belong to an unspecified course experiment; the exact split, preprocessing, and results have not been reproduced here. They are not guaranteed outputs of this lesson’s code.

If the negative-class share in an evaluation set is 0.726, an all-zero classifier achieves 72.6% accuracy and zero recall for positive cases. The reported difference is 83.6−72.6=11.0 percentage points. Relative error reduction would be (27.4−16.4)/27.4≈40.1%, assuming directly comparable evaluations. Neither calculation establishes statistical significance or practical usefulness.

Choose the majority class from training data, then evaluate that fixed prediction on validation and test data. Compare logistic regression and a suitable tree-based model as stronger baselines. On a small medical table, beating majority accuracy is only a first check; calibration, sensitivity, specificity, and external validation matter too.

7. Prepare a table without leaking information

Numeric category codes are not automatically continuous measurements. Chest-pain codes, for example, represent categories; treating them as distances can impose a misleading ordering. One-hot encoding gives each learned category its own indicator column.

Fit imputation, category vocabularies, and numerical scaling on training data only. Apply those fitted transformations to validation and test rows. Avoid calling get_dummies() independently on each split: columns may differ in presence or ordering. Explicit float arrays also avoid mixed object dtypes when strings and numbers coexist.

For a continuous feature, standardization uses x̃=(x−μ_train)/s_train. An unseen validation category must have a defined policy. The example encoder ignores unknown categories, producing zeros for that feature’s indicators; monitor such cases in real use.

Complete example: split, preprocess, train, and evaluate

Run the following blocks in order in a separate Python environment with compatible Keras 3, TensorFlow, pandas, NumPy, and scikit-learn installed. For example, install these packages with python -m pip install keras tensorflow pandas numpy scikit-learn. CPU execution is sufficient; use TensorFlow’s installation guide for platform-specific requirements. This is lesson code, not a dependency change to the website.

The example is designed to be runnable, but its TensorFlow training results are not measured in this lecture. Record package versions and the downloaded file’s checksum when reproducing an experiment.

import os
os.environ["KERAS_BACKEND"] = "tensorflow"

import numpy as np
import pandas as pd
import keras
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import (
    accuracy_score, average_precision_score, confusion_matrix,
    log_loss, precision_score, recall_score, roc_auc_score,
)

keras.utils.set_random_seed(42)
url = "https://storage.googleapis.com/download.tensorflow.org/data/heart.csv"
df = pd.read_csv(url, na_values=["?"])
numeric = ["age", "trestbps", "chol", "thalach", "oldpeak"]
categorical = ["sex", "cp", "fbs", "restecg", "exang", "slope", "ca", "thal"]
assert set(numeric + categorical + ["target"]).issubset(df.columns)
assert not df["target"].isna().any()
assert set(df["target"].unique()) == {0, 1}

X = df[numeric + categorical].copy()
y = df["target"].astype("int32")
X_trainval, X_test, y_trainval, y_test = train_test_split(
    X, y, test_size=0.20, stratify=y, random_state=42,
)
X_train, X_val, y_train, y_val = train_test_split(
    X_trainval, y_trainval, test_size=0.25,
    stratify=y_trainval, random_state=42,
)

numeric_pipe = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
])
category_pipe = Pipeline([
    ("impute", SimpleImputer(strategy="most_frequent")),
    ("encode", OneHotEncoder(handle_unknown="ignore", sparse_output=False)),
])
preprocessor = ColumnTransformer([
    ("numeric", numeric_pipe, numeric),
    ("category", category_pipe, categorical),
])
x_train = preprocessor.fit_transform(X_train).astype("float32")
x_val = preprocessor.transform(X_val).astype("float32")
x_test = preprocessor.transform(X_test).astype("float32")
y_train_array = y_train.to_numpy(dtype="float32").reshape(-1, 1)
y_val_array = y_val.to_numpy(dtype="float32").reshape(-1, 1)
y_test_array = y_test.to_numpy(dtype="float32").reshape(-1, 1)
assert np.isfinite(x_train).all()
assert np.isfinite(x_val).all() and np.isfinite(x_test).all()
print("Split shapes:", x_train.shape, x_val.shape, x_test.shape)

The split is approximately 60% training, 20% validation, and 20% test, with rounding to whole rows. Stratification approximately preserves label proportions; it does not solve duplicate-patient leakage. If multiple rows belong to the same person, use a group-aware split. This code assumes independent patient rows.

The output width depends on category vocabularies observed in training data. 29 inputs describes a particular encoding, not a property of the raw dataset. Derive the width from x_train.shape[1]. The preprocessor already returns NumPy arrays; .to_numpy() is used for pandas objects such as the labels. Conversion alone does not encode strings or handle missing values.

n_features = x_train.shape[1]
inputs = keras.Input(shape=(n_features,), name="features")
h = keras.layers.Dense(16, activation="relu", name="hidden")(inputs)
outputs = keras.layers.Dense(1, activation="sigmoid", name="probability")(h)
model = keras.Model(inputs, outputs)
model.compile(
    optimizer=keras.optimizers.Adam(learning_rate=1e-3),
    loss=keras.losses.BinaryCrossentropy(),
    metrics=[keras.metrics.BinaryAccuracy(name="accuracy")],
)
model.summary()
early_stop = keras.callbacks.EarlyStopping(
    monitor="val_loss", mode="min", patience=15,
    min_delta=1e-4, restore_best_weights=True,
)
history = model.fit(
    x_train, y_train_array,
    validation_data=(x_val, y_val_array),
    epochs=200, batch_size=32, shuffle=True,
    callbacks=[early_stop], verbose=2,
)

compile() configures the training behavior; fit() runs it. epochs=200 is a maximum, not a promise to train for exactly 200 epochs. The callback monitors validation loss and restores its selected best weights when training ends. Its patience and minimum improvement are hyperparameters, not universal recommendations.

An explicit validation set makes split membership visible. With array inputs, validation_split=0.2 instead reserves the last fraction of supplied rows before training shuffling; it does not perform a stratified random split for you. This can be misleading when rows are sorted by outcome or time. Never use the held-out test set as validation_data.

8. Evaluate predictions against the baseline

Once architecture, preprocessing, and any decision threshold are fixed using validation data, evaluate on the test set. The following block uses a threshold of 0.5, chosen in advance, and prints results rather than assuming a target accuracy.

test_metrics = model.evaluate(x_test, y_test_array, return_dict=True, verbose=0)
p_test = model.predict(x_test, verbose=0).ravel()
pred_test = (p_test >= 0.5).astype("int32")
majority_class = int(y_train.mode().iloc[0])
baseline_pred = np.full(len(y_test), majority_class)
baseline_probability = np.full(len(y_test), y_train.mean())

print("Keras test metrics:", test_metrics)
print("Majority accuracy:", accuracy_score(y_test, baseline_pred))
print("Constant-probability log loss:", log_loss(y_test, baseline_probability))
print("NN accuracy:", accuracy_score(y_test, pred_test))
print("NN log loss:", log_loss(y_test, p_test))
print("NN ROC AUC:", roc_auc_score(y_test, p_test))
print("NN average precision:", average_precision_score(y_test, p_test))
print("NN precision:", precision_score(y_test, pred_test, zero_division=0))
print("NN recall:", recall_score(y_test, pred_test, zero_division=0))
print("Confusion matrix [rows=true, columns=predicted]:")
print(confusion_matrix(y_test, pred_test, labels=[0, 1]))

The confusion matrix is [[TN,FP],[FN,TP]]. Sensitivity or recall is TP/(TP+FN); specificity is TN/(TN+FP). ROC AUC measures ranking across thresholds, while average precision summarizes precision–recall behavior. Neither directly establishes that predicted probabilities are calibrated. Assess calibration separately, and interpret every metric in the context of sample size and error costs.

The majority classifier tests a simple decision rule. The constant-probability baseline, using training prevalence, tests probability prediction through log loss. They answer different questions.

A small test sample has substantial uncertainty. If 51 of 61 cases are classified correctly, accuracy is 83.61%. An approximate 95% Wilson interval is roughly 72.4%–90.8%, assuming independent observations. That illustrates uncertainty rather than reproducing the course result. Compare models on the same held-out cases, use repeated or nested cross-validation for development when appropriate, and seek genuinely external data before making claims about other populations.

9. Diagnose the training run

Inspect history.history["loss"] and history.history["val_loss"] across epochs. Both high may indicate underfitting or a broken data pipeline; falling training loss with rising validation loss suggests overfitting or distribution mismatch. Non-finite losses call for checking inputs, numerical stability, and the learning rate.

Early stopping is a first regularization control. Smaller networks, weight penalties, or dropout may help after the pipeline is correct. A dropout layer could be inserted between the hidden representation and output, but its rate must be validated; blindly applying 50% dropout may remove too much useful signal in a small model.

Fix randomness for repeatability, but expect some variation across hardware and software. Keep the feature list, category order, label meaning, preprocessor, model, and decision threshold together. Saving only the neural-network weights is insufficient when preprocessing occurs outside the model. A future application must apply the fitted transformation rather than refitting it on each prediction batch.

10. Exercises with answers

1. Output and loss: How would you change the model to predict a continuous measurement? Use an appropriate output, often Dense(1) with linear activation, and a regression loss such as MSE. Reconsider scaling and target constraints.
2. Shape: Why is Input(shape=29) unsuitable? Keras expects a shape tuple; use (29,), excluding the batch dimension.
3. Capacity: How many parameters are in a 29→16→8→1 dense network? 480+136+9=625.
4. Updates: With 180 training rows, batch size 32, and 10 full epochs, how many updates occur? ceil(180/32)×10=60, retaining the last batch.
5. Loss: For y=0 and p=0.8, what is cross-entropy? −log(0.2)≈1.6094. A high positive probability is incorrect here.
6. Leakage: Why not standardize using all rows before splitting? Held-out values influence the fitted transformation. Learn its statistics from training data only.
7. Early stopping: Validation loss is 0.45, 0.40, 0.38, 0.39, 0.41. With patience two and zero minimum improvement, when does training stop? After epoch 5, restoring epoch 3 when best-weight restoration is enabled.
8. Baseline: In 100 cases, 73 have label zero. What does an all-zero classifier score? 73% accuracy, but zero positive recall, so accuracy alone conceals the failure to detect positive cases.
9. Interpretation: Does 83.6% accuracy prove a model can predict disease one year ahead? No. The target must actually measure that outcome, and validation must match the intended future use.

Further reading

• Keras Input API: shapes and symbolic inputs.
• Keras Functional API guide: graph-based model construction.
• Keras model training APIs: compile, fit, evaluate, and validation splitting.
• Keras EarlyStopping: patience, minimum improvement, and restoring weights.
• TensorFlow automatic differentiation: gradients and gradient tapes.
• TensorFlow installation guide: environment and device requirements.
• scikit-learn ColumnTransformer and OneHotEncoder: fitted transformations and category handling.

The examples are educational. Dataset documentation establishes the label meaning; the supplied course percentages remain attributed reports rather than independently verified results or a clinical performance claim.

Feynman study lab

Explain it simply

Training code is only one part of a model; data preparation and evaluation must match how it will be used.

Concrete example. Estimate a numeric feature’s mean using training rows, then use that same mean to transform validation rows.

Teach it back without looking

Close the lesson and explain the idea aloud to a beginner in two minutes. Define each technical term you use, walk through the example, and state one assumption or limitation. Then answer:

Why should you not fit preprocessing on the entire dataset before splitting?

Check your explanation after you try

Validation information would influence training preparation. Fit transformations on training data and apply them consistently to held-out data.

Find the gap and try again

If you hesitated, relied on a formula you could not explain, or missed the self-check, write down the exact gap. Revisit the relevant section of this lesson and work through its example. Close the notes, explain it again in simpler language, then change one number or assumption and predict what changes. Recalculate or check the lesson to test your prediction. Try once more tomorrow from memory.