❇️Understanding Data Types in Pandas

⏺️Why Are Data Types Important?

When working with datasets in Pandas, each column has a specific data type that defines how the data is stored and what operations can be performed on it. Incorrect data types can lead to errors, slow performance, and incorrect results.

For example:

• If numeric data is stored as text (object), mathematical operations won’t work correctly.

• If dates are stored as strings, sorting and filtering by date won’t function properly.

• Using categorical data types instead of text can improve memory efficiency.

Understanding and correctly setting data types is a crucial step in data cleaning, preprocessing, and analysis. Let's explore how to check, interpret, and convert data types effectively using Pandas. 🚀

⏺️Checking data types in a Dataframe

To check the data types of all columns:

import pandas as pd

# Load a dataset
df = pd.read_csv("your_dataset.csv")

# Check data types
print(df.dtypes)

Example Output:

ID               int64
Name            object
Age             float64
Salary          float64
Join_Date       object
Department      object
dtype: object

Each column belongs to one of the main data types in Pandas.

⏺️Common Pandas Data Types

❇️Real-World Issues Due to Wrong Data Types & Solutions

1️⃣Numbers Stored as strings (Object type)

🔥 Problem:

Imagine a dataset with salaries, but they are stored as text:

import pandas as pd

data = {'Employee': ['Anushka', 'Benstokes', 'Chahal'],
        'Salary': ['50000', '60000', '55000']}  # Incorrect: Stored as strings

df = pd.DataFrame(data)  # This creates a DataFrame in memory. It doesnot store it as a csv file.
                         # To save it as csv file, you need to use
                         # df.to_csv("filename.csv", index = False)
# Index colomns are leftmost colomn that uniquely each row in a dataframe.
print(df.dtypes)

# OUTPUT
Employee    object
Salary      object  ❌ (Should be numeric)
dtype: object

Since Salary is stored as a string (object) instead of a numeric type, calculations won’t work:

print(df["Salary"].mean())  # ❌ Error: Cannot calculate mean on strings

✔ Fix: Convert it to numeric:

df["Salary"] = pd.to_numeric(df["Salary"])

NOTE : DataFrame() in Pandas is a two-dimensional, tabular data structure similar to an Excel spreadsheet or SQL table. It consists of rows and columns, where:

✅ Rows have index labels
✅ Columns have names and data types
✅ Supports heterogeneous data types

NOTE : df is not a file, but a Pandas DataFrame object stored in memory. It represents tabular data, like an Excel sheet, but it exists only in your Python session unless you explicitly save it to a file (e.g., CSV, Excel, or SQL).

2️⃣ Dates Stored as Strings

🔥 Problem:

A dataset contains dates, but they are stored as text (object) instead of datetime:

data = {'Name': ['Anushka', 'Benstokes', 'Chahal'],
        'Join_Date': ['2024-02-10', '2023-05-22', '2022-11-15']}  # Stored as strings

df = pd.DataFrame(data)  
print(df.dtypes)

Name         object
Join_Date    object  ❌ (Should be datetime)
dtype: object

If you try to sort the dates:

df.sort_values(by="Join_Date")

It will sort alphabetically, not by actual date order!

✔ Fix: Convert to datetime:

df["Join_Date"] = pd.to_datetime(df["Join_Date"])

Now, sorting works correctly.

3️⃣ Floating-Point Numbers Stored as Integers

🔥 Problem:

Suppose you have a Price column, but it's stored as an integer (int64) instead of a float (float64):

data = {'Item': ['Apple', 'Banana', 'Cherry'],
        'Price': [100, 50, 75]}  # Stored as int, but prices may need decimal places

df = pd.DataFrame(data)
print(df.dtypes)

Item      object
Price     int64  ❌ (Should be float64)
dtype: object

If you later divide prices by 3, the result won’t retain decimals:

df["Price"] = df["Price"] / 3
print(df["Price"])

✔ Fix: Convert int64 to float64:

df["Price"] = df["Price"].astype("float64")

4️⃣ Categorical Data Stored as Object

🔥 Problem:

A dataset contains a Department column with repeated values. Storing it as object wastes memory and slows down operations:

categorical variable refers to a column that contains a limited number of unique values (categories) instead of continuous numerical values.

data = {'Employee': ['Anushka', 'Benstokes', 'Chahal'],
        'Department': ['HR', 'Finance', 'HR']}  # Stored as object

df = pd.DataFrame(data)
print(df.dtypes)

Employee      object
Department    object  ❌ (Better as category)
dtype: object

Since departments are limited in number, storing them as category reduces memory usage and speeds up processing.

✔ Fix: Convert to category:

df["Department"] = df["Department"].astype("category")

❇️Let’s Explore More !!

Let's intentionally create a CSV file with errors, missing data, outliers, duplicates, and inconsistencies so that we can analyze and fix these issues step by step. With this example , you will be to understand this more properly.

📌 Step 1: Creating a CSV File with Errors

We'll generate an Employee dataset (error_employees.csv) that includes:
✅ Missing values in Age, Department, Salary
✅ Outliers in Salary (e.g., an extremely high value)
✅ Incorrect data types (e.g., Join_Date as text)
✅ Duplicate rows
✅ Inconsistent department names (e.g., IT, I.T, it)

🔹 Creating the CSV File

import pandas as pd

# Defining the dataset with errors
data = {
    "Employee_ID": [101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 103, 108],  
    "Name": ["Alice", "Bob", "Charlie", "David", "Emma", "Frank", "Grace", "Henry", "Ivy", "Jack", "Charlie", "Henry"],  
    "Age": [29, None, 40, 50, 28, -5, None, 36, 999, None, 40, 36],  # Missing & unrealistic values  
    "Department": ["HR", "IT", "Finance", "I.T", "Marketing", "HR", None, "Finance", "it", "Sales", "Finance", "Finance"],  
    "Salary": [50000, 70000, 90000, None, 60000, 45000, 100000000, 95000, -30000, 55000, 90000, 95000],  # Outliers & negative values  
    "Join_Date": ["2018-06-23", "Not Available", "2013-11-01", "2010-08-12", "2019-04-25",  
                  "2021-01-30", "2016-12-05", "2012-07-19", "2014-05-14", "2020-03-21",  
                  "2013-11-01", "2012-07-19"]  # Incorrect date format  
}

# Creating the DataFrame
df = pd.DataFrame(data)

# Saving as CSV
df.to_csv("error_employees.csv", index=False)

print("CSV file 'error_employees.csv' with errors created successfully!")

Now, we will analyze and clean this dataset step by step. We have created our own dataset just for this article purpose. There is no need to create dataset in this way. You can always download any dataset that is available freely on internet and work on that in the next following way.

Our csv file will looked like this

Employee_ID,Name,Age,Department,Salary,Join_Date
101,Alice,29,HR,50000,2018-06-23
102,Bob,,IT,70000,Not Available
103,Charlie,40,Finance,90000,2013-11-01
104,David,50,I.T,,2010-08-12
105,Emma,28,Marketing,60000,2019-04-25
106,Frank,-5,HR,45000,2021-01-30
107,Grace,, ,100000000,2016-12-05
108,Henry,36,Finance,95000,2012-07-19
109,Ivy,999,it,-30000,2014-05-14
110,Jack,,Sales,55000,2020-03-21
103,Charlie,40,Finance,90000,2013-11-01
108,Henry,36,Finance,95000,2012-07-19

🔍 Step 2: Data Analysis & Cleaning

🔹 1. Checking the Structure of Data

Why?
Before analysis, we need to check how many rows and columns exist.

print(df.shape)

📌 Output: (12, 6) → 12 rows & 6 columns

print(df.dtypes)

📌 Output:

Employee_ID      int64  
Name           object  
Age           float64  
Department     object  
Salary        float64  
Join_Date      object

Now seeing the data types of each respective column, we can identify some issues

• Join_Date should be a datetime format, but it’s stored as text.

• Age may contain negative & unrealistic values as it is stored as float data type