# Data Inspection and Preprocessing with Pandas

# ❇️Understanding Data Types in Pandas

### **⏺️Why Are Data Types Important?**

When working with datasets in **Pandas**, each column has a specific **data type** that defines how the data is stored and what operations can be performed on it. **Incorrect data types can lead to errors, slow performance, and incorrect results.**

For example:

* If numeric data is stored as **text (object)**, mathematical operations won’t work correctly.
    
* If dates are stored as **strings**, sorting and filtering by date won’t function properly.
    
* Using **categorical** data types instead of text can improve memory efficiency.
    

Understanding and correctly setting **data types** is a **crucial step** in **data cleaning, preprocessing, and analysis**. Let's explore how to **check, interpret, and convert** data types effectively using Pandas. 🚀

### ⏺️Checking data types in a Dataframe

To check the data types of all columns:

```python
import pandas as pd

# Load a dataset
df = pd.read_csv("your_dataset.csv")

# Check data types
print(df.dtypes)
```

Example Output:

```css
ID               int64
Name            object
Age             float64
Salary          float64
Join_Date       object
Department      object
dtype: object
```

Each column belongs to one of the **main data types** in Pandas.

### ⏺️Common Pandas Data Types

| **<mark>Data Types</mark>** | **<mark>Description</mark>** |
| --- | --- |
| `int64` | Integer values (whole numbers) |
| `float64` | Decimal numbers |
| `object` | Text/string data |
| `bool` | Boolean values (`True` / `False`) |
| `datetime64` | Date and time values |
| `category` | Categorical data (fixed number of categories) |

# ❇️Real-World Issues Due to Wrong Data Types & Solutions

### ***1️⃣Numbers Stored as strings (Object type)***

#### 🔥 **Problem:**

Imagine a dataset with salaries, but they are stored as text:

```python
import pandas as pd

data = {'Employee': ['Anushka', 'Benstokes', 'Chahal'],
        'Salary': ['50000', '60000', '55000']}  # Incorrect: Stored as strings

df = pd.DataFrame(data)  # This creates a DataFrame in memory. It doesnot store it as a csv file.
                         # To save it as csv file, you need to use
                         # df.to_csv("filename.csv", index = False)
# Index colomns are leftmost colomn that uniquely each row in a dataframe.
print(df.dtypes)
```

```python
# OUTPUT
Employee    object
Salary      object  ❌ (Should be numeric)
dtype: object
```

Since `Salary` is stored as a **string (object)** instead of a **numeric type**, calculations won’t work:

```python
print(df["Salary"].mean())  # ❌ Error: Cannot calculate mean on strings
```

**✔ Fix:** Convert it to numeric:

```python
df["Salary"] = pd.to_numeric(df["Salary"])
```

***<mark>NOTE </mark>* <mark>: </mark>** `DataFrame()` in Pandas is a **two-dimensional, tabular data structure** similar to an Excel spreadsheet or SQL table. It consists of **rows and columns**, where:

✅ Rows have **index labels**  
✅ Columns have **names and data types**  
✅ Supports **heterogeneous data types**

***<mark>NOTE </mark>* <mark>: </mark>** `df` is **not a file**, but a **Pandas DataFrame object** stored in memory. It represents tabular data, like an Excel sheet, but it exists only in your Python session unless you explicitly **save** it to a file (e.g., CSV, Excel, or SQL).

### ***2️⃣ Dates Stored as Strings***

#### 🔥 **Problem:**

A dataset contains dates, but they are stored as **text (object)** instead of `datetime`:

```python
data = {'Name': ['Anushka', 'Benstokes', 'Chahal'],
        'Join_Date': ['2024-02-10', '2023-05-22', '2022-11-15']}  # Stored as strings

df = pd.DataFrame(data)  
print(df.dtypes)
```

```css
Name         object
Join_Date    object  ❌ (Should be datetime)
dtype: object
```

If you try to sort the dates:

```python
df.sort_values(by="Join_Date")
```

It will **sort alphabetically**, not by actual date order!

**✔ Fix:** Convert to `datetime`:

```python
df["Join_Date"] = pd.to_datetime(df["Join_Date"])
```

Now, sorting works correctly.

### ***3️⃣ Floating-Point Numbers Stored as Integers***

#### 🔥 **Problem:**

Suppose you have a `Price` column, but it's stored as an **integer (**`int64`) instead of a **float (**`float64`):

```python
data = {'Item': ['Apple', 'Banana', 'Cherry'],
        'Price': [100, 50, 75]}  # Stored as int, but prices may need decimal places

df = pd.DataFrame(data)
print(df.dtypes)
```

```css
Item      object
Price     int64  ❌ (Should be float64)
dtype: object
```

If you later divide prices by 3, the result won’t retain decimals:

```python
df["Price"] = df["Price"] / 3
print(df["Price"])
```

**✔ Fix:** Convert `int64` to `float64`:

```python
df["Price"] = df["Price"].astype("float64")
```

### ***4️⃣*** Categorical Data Stored as Object

#### 🔥 **Problem:**

A dataset contains a **Department** column with repeated values. Storing it as `object` **wastes memory** and slows down operations:

> **categorical variable** refers to a column that contains a **limited number of unique values (categories)** instead of continuous numerical values.

```python
data = {'Employee': ['Anushka', 'Benstokes', 'Chahal'],
        'Department': ['HR', 'Finance', 'HR']}  # Stored as object

df = pd.DataFrame(data)
print(df.dtypes)
```

```css
Employee      object
Department    object  ❌ (Better as category)
dtype: object
```

Since **departments are limited in number**, storing them as **category** reduces memory usage and speeds up processing.

**✔ Fix:** Convert to `category`:

```css
df["Department"] = df["Department"].astype("category")
```

# ❇️Let’s Explore More !!

Let's **intentionally create a CSV file** with **errors, missing data, outliers, duplicates, and inconsistencies** so that we can analyze and fix these issues step by step. With this example , you will be to understand this more properly.

## 📌 Step 1: Creating a CSV File with Errors

We'll generate an **Employee dataset (**`error_employees.csv`) that includes:  
✅ **Missing values** in `Age`, `Department`, `Salary`  
✅ **Outliers** in `Salary` (e.g., an extremely high value)  
✅ **Incorrect data types** (e.g., `Join_Date` as text)  
✅ **Duplicate rows**  
✅ **Inconsistent department names** (e.g., `IT`, `I.T`, `it`)

**🔹 Creating the CSV File**

```python
import pandas as pd

# Defining the dataset with errors
data = {
    "Employee_ID": [101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 103, 108],  
    "Name": ["Alice", "Bob", "Charlie", "David", "Emma", "Frank", "Grace", "Henry", "Ivy", "Jack", "Charlie", "Henry"],  
    "Age": [29, None, 40, 50, 28, -5, None, 36, 999, None, 40, 36],  # Missing & unrealistic values  
    "Department": ["HR", "IT", "Finance", "I.T", "Marketing", "HR", None, "Finance", "it", "Sales", "Finance", "Finance"],  
    "Salary": [50000, 70000, 90000, None, 60000, 45000, 100000000, 95000, -30000, 55000, 90000, 95000],  # Outliers & negative values  
    "Join_Date": ["2018-06-23", "Not Available", "2013-11-01", "2010-08-12", "2019-04-25",  
                  "2021-01-30", "2016-12-05", "2012-07-19", "2014-05-14", "2020-03-21",  
                  "2013-11-01", "2012-07-19"]  # Incorrect date format  
}

# Creating the DataFrame
df = pd.DataFrame(data)

# Saving as CSV
df.to_csv("error_employees.csv", index=False)

print("CSV file 'error_employees.csv' with errors created successfully!")
```

Now, we will **analyze and clean this dataset step by step**. We have created our own dataset just for this article purpose. There is no need to create dataset in this way. You can always download any dataset that is available freely on internet and work on that in the next following way.

Our csv file will looked like this

```python
Employee_ID,Name,Age,Department,Salary,Join_Date
101,Alice,29,HR,50000,2018-06-23
102,Bob,,IT,70000,Not Available
103,Charlie,40,Finance,90000,2013-11-01
104,David,50,I.T,,2010-08-12
105,Emma,28,Marketing,60000,2019-04-25
106,Frank,-5,HR,45000,2021-01-30
107,Grace,, ,100000000,2016-12-05
108,Henry,36,Finance,95000,2012-07-19
109,Ivy,999,it,-30000,2014-05-14
110,Jack,,Sales,55000,2020-03-21
103,Charlie,40,Finance,90000,2013-11-01
108,Henry,36,Finance,95000,2012-07-19
```

## **🔍 Step 2: Data Analysis & Cleaning**

### 🔹 1. Checking the Structure of Data

**Why?**  
Before analysis, we need to check **how many rows and columns** exist.

```python
print(df.shape)
```

📌 **Output:** `(12, 6)` → 12 rows & 6 columns

```python
print(df.dtypes)
```

📌 **Output:**

```python
Employee_ID      int64  
Name           object  
Age           float64  
Department     object  
Salary        float64  
Join_Date      object
```

Now seeing the data types of each respective column, we can identify some issues

* `Join_Date` should be **a datetime format, but it’s stored as text**.
    
* `Age` may contain **negative & unrealistic values as it is stored as float data type**