Data Science

Data leakage happens when information that would not be available at prediction time influences model training. The result is an evaluation score that looks excellent in development and collapses after deployment. Leakage is often subtle because the model code itself can be correct. The mistake lives in how datasets, features, preprocessing, and time boundaries are constructed. Split before learning from the data A classic mistake is standardizing the full dataset and splitting afterward.

A classifier can rank examples correctly while producing probabilities that are poor estimates of real-world likelihood. If a model assigns 0.8 probability to many comparable cases, calibration asks whether roughly 80% of those cases are actually positive. This matters whenever probabilities drive decisions such as pricing, triage, alert thresholds, expected value, or human review. Discrimination and calibration are different Metrics such as ROC AUC evaluate how well a model ranks positive examples above negative ones. They do not require predicted probabilities to match observed frequencies.

CSV (Comma-Separated Values) is a common format for storing and exchanging tabular data. Pandas makes it straightforward to export a DataFrame to CSV and load CSV data back into a DataFrame. Install Pandas If Pandas is not installed yet: Copy python -m pip install pandas Using python -m pip helps ensure that pip belongs to the Python interpreter you intend to use.

Pandas is a widely used Python library for tabular data manipulation and analysis. This guide covers a few everyday DataFrame operations: renaming columns, adding and updating rows, deleting data, sorting, and filtering. Install Pandas Copy python -m pip install pandas Create a DataFrame Copy import pandas as pd df = pd.DataFrame( { "Name": [ "Braund, Mr. Owen Harris", "Allen, Mr. William Henry", "Bonnell, Miss. Elizabeth", ], "Age": [22, 35, 58], "Sex": ["male", "male", "female"], } ) Rename a Column Copy df = df.rename(columns={"Sex": "Gender"}) Returning a new DataFrame instead of relying on inplace=True often makes transformation pipelines easier to reason about.