Data Science on Nalar

Avoiding Data Leakage in Machine Learning Pipelines

Tue, 01 Sep 2026 00:00:00 +0700

Data leakage happens when information that would not be available at prediction time influences model training. The result is an evaluation score that looks excellent in development and collapses after deployment.

Leakage is often subtle because the model code itself can be correct. The mistake lives in how datasets, features, preprocessing, and time boundaries are constructed.

Split before learning from the data

A classic mistake is standardizing the full dataset and splitting afterward.

Probability Calibration for Classification Models

Tue, 01 Sep 2026 00:00:00 +0700

A classifier can rank examples correctly while producing probabilities that are poor estimates of real-world likelihood.

If a model assigns 0.8 probability to many comparable cases, calibration asks whether roughly 80% of those cases are actually positive. This matters whenever probabilities drive decisions such as pricing, triage, alert thresholds, expected value, or human review.

Discrimination and calibration are different

Metrics such as ROC AUC evaluate how well a model ranks positive examples above negative ones. They do not require predicted probabilities to match observed frequencies.

Working with CSV Files in Pandas

Wed, 21 Aug 2024 00:00:00 +0700

CSV (Comma-Separated Values) is a common format for storing and exchanging tabular data. Pandas makes it straightforward to export a DataFrame to CSV and load CSV data back into a DataFrame.

Install Pandas

If Pandas is not installed yet:

$el.innerText = 'Copy', 2000)" class="absolute top-2 right-2 bg-neutral-700 text-white text-xs px-2 py-1 rounded opacity-0 group-hover:opacity-100 transition-opacity" > Copy

python -m pip install pandas

Using python -m pip helps ensure that pip belongs to the Python interpreter you intend to use.

Working with Pandas: A Beginner Guide

Wed, 21 Aug 2024 00:00:00 +0700

Pandas is a widely used Python library for tabular data manipulation and analysis. This guide covers a few everyday DataFrame operations: renaming columns, adding and updating rows, deleting data, sorting, and filtering.

Install Pandas

$el.innerText = 'Copy', 2000)" class="absolute top-2 right-2 bg-neutral-700 text-white text-xs px-2 py-1 rounded opacity-0 group-hover:opacity-100 transition-opacity" > Copy

python -m pip install pandas

Create a DataFrame

$el.innerText = 'Copy', 2000)" class="absolute top-2 right-2 bg-neutral-700 text-white text-xs px-2 py-1 rounded opacity-0 group-hover:opacity-100 transition-opacity" > Copy

import pandas as pd

df = pd.DataFrame(
    {
        "Name": [
            "Braund, Mr. Owen Harris",
            "Allen, Mr. William Henry",
            "Bonnell, Miss. Elizabeth",
        ],
        "Age": [22, 35, 58],
        "Sex": ["male", "male", "female"],
    }
)

Rename a Column

$el.innerText = 'Copy', 2000)" class="absolute top-2 right-2 bg-neutral-700 text-white text-xs px-2 py-1 rounded opacity-0 group-hover:opacity-100 transition-opacity" > Copy

df = df.rename(columns={"Sex": "Gender"})

Returning a new DataFrame instead of relying on inplace=True often makes transformation pipelines easier to reason about.