Python for Data Science: A Complete Guide to ML and Analytics

Jan 20, 2024 • 12 min read

Python has become the de facto language for data science and machine learning. With its rich ecosystem of libraries and tools, Python provides everything needed for data analysis, visualization, and building machine learning models. This comprehensive guide covers the essential tools and techniques for data science with Python.

Essential Python Libraries for Data Science

Core Libraries Overview

  • NumPy: Numerical computing and array operations
  • Pandas: Data manipulation and analysis
  • Matplotlib/Seaborn: Data visualization
  • Scikit-learn: Machine learning algorithms
  • Jupyter: Interactive development environment

Data Manipulation with Pandas

Reading and Writing Data

import pandas as pd
import numpy as np

# Read different file formats
df_csv = pd.read_csv('data.csv')
df_excel = pd.read_excel('data.xlsx')
df_json = pd.read_json('data.json')
df_sql = pd.read_sql('SELECT * FROM table', connection)

# Write data to files
df.to_csv('output.csv', index=False)
df.to_excel('output.xlsx', sheet_name='Sheet1')
df.to_json('output.json', orient='records')

Data Exploration and Cleaning

# Basic data exploration
print(df.info())
print(df.describe())
print(df.head())
print(df.shape)

# Check for missing values
print(df.isnull().sum())
print(df.isnull().sum() / len(df) * 100)

# Handle missing values
df_cleaned = df.dropna()  # Remove rows with missing values
df_filled = df.fillna(df.mean())  # Fill with mean
df_interpolated = df.interpolate()  # Interpolate missing values

# Remove duplicates
df_unique = df.drop_duplicates()

# Data types conversion
df['date'] = pd.to_datetime(df['date'])
df['category'] = df['category'].astype('category')

Advanced Data Manipulation

# Filtering data
filtered_df = df[(df['age'] > 25) & (df['salary'] > 50000)]

# Grouping and aggregation
grouped = df.groupby('department').agg({
    'salary': ['mean', 'std', 'count'],
    'age': 'mean'
})

# Pivot tables
pivot_table = df.pivot_table(
    values='salary',
    index='department',
    columns='gender',
    aggfunc='mean'
)

# Merging dataframes
merged_df = pd.merge(df1, df2, on='id', how='inner')
concatenated_df = pd.concat([df1, df2], axis=0)

# Apply functions
df['salary_category'] = df['salary'].apply(
    lambda x: 'High' if x > 75000 else 'Medium' if x > 50000 else 'Low'
)

# Vectorized operations
df['salary_increase'] = df['salary'] * 1.1
df['full_name'] = df['first_name'] + ' ' + df['last_name']

Numerical Computing with NumPy

Array Operations

import numpy as np

# Create arrays
arr1 = np.array([1, 2, 3, 4, 5])
arr2 = np.zeros((3, 4))
arr3 = np.ones((2, 3))
arr4 = np.arange(0, 10, 2)
arr5 = np.linspace(0, 1, 5)

# Array operations
arr_squared = arr1 ** 2
arr_sum = np.sum(arr1)
arr_mean = np.mean(arr1)
arr_std = np.std(arr1)

# Broadcasting
arr_2d = np.array([[1, 2, 3], [4, 5, 6]])
arr_2d + 10  # Broadcasting

# Random numbers
random_arr = np.random.randn(1000)
normal_dist = np.random.normal(0, 1, 1000)

Linear Algebra

# Matrix operations
A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])

# Matrix multiplication
C = np.dot(A, B)
C_alt = A @ B

# Eigenvalues and eigenvectors
eigenvalues, eigenvectors = np.linalg.eig(A)

# Solving linear equations
# Ax = b
A = np.array([[2, 1], [1, 3]])
b = np.array([4, 5])
x = np.linalg.solve(A, b)

Data Visualization

Matplotlib Basics

import matplotlib.pyplot as plt
import seaborn as sns

# Set style
plt.style.use('seaborn-v0_8')
sns.set_palette("husl")

# Line plot
plt.figure(figsize=(10, 6))
plt.plot(df['date'], df['value'], marker='o')
plt.title('Time Series Plot')
plt.xlabel('Date')
plt.ylabel('Value')
plt.xticks(rotation=45)
plt.grid(True)
plt.tight_layout()
plt.show()

# Scatter plot
plt.figure(figsize=(8, 6))
plt.scatter(df['x'], df['y'], alpha=0.6)
plt.title('Scatter Plot')
plt.xlabel('X')
plt.ylabel('Y')
plt.show()

# Histogram
plt.figure(figsize=(8, 6))
plt.hist(df['salary'], bins=30, alpha=0.7, edgecolor='black')
plt.title('Salary Distribution')
plt.xlabel('Salary')
plt.ylabel('Frequency')
plt.show()

Advanced Visualizations with Seaborn

# Distribution plots
fig, axes = plt.subplots(2, 2, figsize=(12, 10))

# Histogram with KDE
sns.histplot(data=df, x='salary', kde=True, ax=axes[0, 0])
axes[0, 0].set_title('Salary Distribution')

# Box plot
sns.boxplot(data=df, x='department', y='salary', ax=axes[0, 1])
axes[0, 1].set_title('Salary by Department')

# Violin plot
sns.violinplot(data=df, x='department', y='age', ax=axes[1, 0])
axes[1, 0].set_title('Age Distribution by Department')

# Correlation heatmap
correlation_matrix = df[['salary', 'age', 'experience']].corr()
sns.heatmap(correlation_matrix, annot=True, cmap='coolwarm', ax=axes[1, 1])
axes[1, 1].set_title('Correlation Matrix')

plt.tight_layout()
plt.show()

# Pair plot
sns.pairplot(df[['salary', 'age', 'experience']], diag_kind='kde')
plt.show()

# Joint plot
sns.jointplot(data=df, x='age', y='salary', kind='reg')
plt.show()

Machine Learning with Scikit-learn

Data Preprocessing

from sklearn.preprocessing import StandardScaler, LabelEncoder, OneHotEncoder
from sklearn.model_selection import train_test_split
from sklearn.compose import ColumnTransformer

# Split features and target
X = df.drop('target', axis=1)
y = df['target']

# Split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Handle categorical variables
categorical_features = ['department', 'gender']
numerical_features = ['age', 'experience', 'salary']

# Create preprocessing pipeline
preprocessor = ColumnTransformer(
    transformers=[
        ('num', StandardScaler(), numerical_features),
        ('cat', OneHotEncoder(drop='first'), categorical_features)
    ]
)

# Apply preprocessing
X_train_processed = preprocessor.fit_transform(X_train)
X_test_processed = preprocessor.transform(X_test)

Supervised Learning

Classification

from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score

# Logistic Regression
lr_model = LogisticRegression(random_state=42)
lr_model.fit(X_train_processed, y_train)
lr_predictions = lr_model.predict(X_test_processed)
lr_probabilities = lr_model.predict_proba(X_test_processed)

# Random Forest
rf_model = RandomForestClassifier(n_estimators=100, random_state=42)
rf_model.fit(X_train_processed, y_train)
rf_predictions = rf_model.predict(X_test_processed)

# Support Vector Machine
svm_model = SVC(probability=True, random_state=42)
svm_model.fit(X_train_processed, y_train)
svm_predictions = svm_model.predict(X_test_processed)

# Evaluate models
print("Logistic Regression Classification Report:")
print(classification_report(y_test, lr_predictions))

print("Random Forest Classification Report:")
print(classification_report(y_test, rf_predictions))

# Confusion Matrix
plt.figure(figsize=(8, 6))
sns.heatmap(confusion_matrix(y_test, lr_predictions),
            annot=True, fmt='d', cmap='Blues')
plt.title('Confusion Matrix')
plt.ylabel('True Label')
plt.xlabel('Predicted Label')
plt.show()

Regression

from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error, r2_score

# Linear Regression
lr_reg = LinearRegression()
lr_reg.fit(X_train_processed, y_train)
lr_predictions = lr_reg.predict(X_test_processed)

# Ridge Regression
ridge_reg = Ridge(alpha=1.0)
ridge_reg.fit(X_train_processed, y_train)
ridge_predictions = ridge_reg.predict(X_test_processed)

# Random Forest Regression
rf_reg = RandomForestRegressor(n_estimators=100, random_state=42)
rf_reg.fit(X_train_processed, y_train)
rf_predictions = rf_reg.predict(X_test_processed)

# Evaluate regression models
models = {
    'Linear Regression': lr_reg,
    'Ridge Regression': ridge_reg,
    'Random Forest': rf_reg
}

for name, model in models.items():
    predictions = model.predict(X_test_processed)
    mse = mean_squared_error(y_test, predictions)
    r2 = r2_score(y_test, predictions)
    print(f"{name}:")
    print(f"  MSE: {mse:.4f}")
    print(f"  R²: {r2:.4f}")
    print()

Unsupervised Learning

Clustering

from sklearn.cluster import KMeans, DBSCAN
from sklearn.decomposition import PCA
from sklearn.metrics import silhouette_score

# K-Means Clustering
kmeans = KMeans(n_clusters=3, random_state=42)
kmeans_labels = kmeans.fit_predict(X_train_processed)

# Find optimal number of clusters
silhouette_scores = []
K_range = range(2, 11)

for k in K_range:
    kmeans = KMeans(n_clusters=k, random_state=42)
    labels = kmeans.fit_predict(X_train_processed)
    score = silhouette_score(X_train_processed, labels)
    silhouette_scores.append(score)

# Plot silhouette scores
plt.figure(figsize=(8, 6))
plt.plot(K_range, silhouette_scores, 'bo-')
plt.xlabel('Number of Clusters')
plt.ylabel('Silhouette Score')
plt.title('Silhouette Score vs Number of Clusters')
plt.grid(True)
plt.show()

# PCA for dimensionality reduction
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_train_processed)

# Visualize clusters
plt.figure(figsize=(10, 6))
scatter = plt.scatter(X_pca[:, 0], X_pca[:, 1],
                     c=kmeans_labels, cmap='viridis')
plt.colorbar(scatter)
plt.title('Clusters in 2D (PCA)')
plt.xlabel('First Principal Component')
plt.ylabel('Second Principal Component')
plt.show()

Dimensionality Reduction

from sklearn.decomposition import PCA
from sklearn.manifold import TSNE

# PCA
pca = PCA(n_components=0.95)  # Keep 95% of variance
X_pca = pca.fit_transform(X_train_processed)
print(f"Original dimensions: {X_train_processed.shape[1]}")
print(f"PCA dimensions: {X_pca.shape[1]}")
print(f"Explained variance ratio: {pca.explained_variance_ratio_.sum():.3f}")

# t-SNE for visualization
tsne = TSNE(n_components=2, random_state=42)
X_tsne = tsne.fit_transform(X_train_processed)

plt.figure(figsize=(10, 6))
plt.scatter(X_tsne[:, 0], X_tsne[:, 1], alpha=0.6)
plt.title('t-SNE Visualization')
plt.xlabel('First t-SNE Component')
plt.ylabel('Second t-SNE Component')
plt.show()

Model Evaluation and Selection

Cross-Validation

from sklearn.model_selection import cross_val_score, GridSearchCV
from sklearn.pipeline import Pipeline

# Create pipeline
pipeline = Pipeline([
    ('preprocessor', preprocessor),
    ('classifier', RandomForestClassifier(random_state=42))
])

# Cross-validation
cv_scores = cross_val_score(pipeline, X, y, cv=5, scoring='accuracy')
print(f"Cross-validation scores: {cv_scores}")
print(f"Mean CV score: {cv_scores.mean():.3f} (+/- {cv_scores.std() * 2:.3f})")

# Hyperparameter tuning
param_grid = {
    'classifier__n_estimators': [50, 100, 200],
    'classifier__max_depth': [None, 10, 20],
    'classifier__min_samples_split': [2, 5, 10]
}

grid_search = GridSearchCV(
    pipeline, param_grid, cv=5, scoring='accuracy', n_jobs=-1
)
grid_search.fit(X, y)

print(f"Best parameters: {grid_search.best_params_}")
print(f"Best cross-validation score: {grid_search.best_score_:.3f}")

Feature Importance

# Feature importance for Random Forest
feature_names = (numerical_features +
                [f"{col}_{val}" for col, vals in
                 preprocessor.named_transformers_['cat'].categories_
                 for val in vals[1:]])

importance_df = pd.DataFrame({
    'feature': feature_names,
    'importance': rf_model.feature_importances_
}).sort_values('importance', ascending=False)

plt.figure(figsize=(10, 6))
sns.barplot(data=importance_df.head(10), x='importance', y='feature')
plt.title('Top 10 Feature Importances')
plt.xlabel('Importance')
plt.show()

Time Series Analysis

import pandas as pd
from statsmodels.tsa.seasonal import seasonal_decompose
from statsmodels.tsa.arima.model import ARIMA
from statsmodels.graphics.tsaplots import plot_acf, plot_pacf

# Time series data
df_time = pd.read_csv('time_series_data.csv')
df_time['date'] = pd.to_datetime(df_time['date'])
df_time.set_index('date', inplace=True)

# Decomposition
decomposition = seasonal_decompose(df_time['value'], period=12)

fig, axes = plt.subplots(4, 1, figsize=(12, 10))
decomposition.observed.plot(ax=axes[0])
axes[0].set_title('Observed')
decomposition.trend.plot(ax=axes[1])
axes[1].set_title('Trend')
decomposition.seasonal.plot(ax=axes[2])
axes[2].set_title('Seasonal')
decomposition.resid.plot(ax=axes[3])
axes[3].set_title('Residual')
plt.tight_layout()
plt.show()

# ARIMA model
model = ARIMA(df_time['value'], order=(1, 1, 1))
model_fit = model.fit()
forecast = model_fit.forecast(steps=12)

Best Practices for Data Science

1. Data Quality

  • Always check for missing values and outliers
  • Validate data types and ranges
  • Document data sources and transformations

2. Model Selection

  • Start with simple models
  • Use cross-validation for model evaluation
  • Consider interpretability vs. performance trade-offs

3. Feature Engineering

  • Create domain-specific features
  • Handle categorical variables appropriately
  • Scale numerical features

4. Visualization

  • Choose appropriate plot types
  • Use consistent color schemes
  • Include proper labels and titles

5. Reproducibility

  • Set random seeds
  • Document preprocessing steps
  • Use version control for code

Conclusion

Python provides a comprehensive ecosystem for data science and machine learning. The combination of pandas for data manipulation, numpy for numerical computing, matplotlib/seaborn for visualization, and scikit-learn for machine learning makes Python the go-to language for data science.

Key takeaways:

  • Master pandas for efficient data manipulation
  • Use appropriate visualizations for different data types
  • Start with simple models and gradually increase complexity
  • Always validate your models with proper evaluation metrics
  • Focus on data quality and preprocessing

Remember that data science is iterative - start with exploration, build simple models, and gradually refine your approach based on results and domain knowledge.