Dataset: Melbourne Housing¶
About dataset: Melbourne Housing Market
Importing Modules¶
In [1]:
import pandas as pd # import modules pandas as pd
Load Dataset¶
In [2]:
housing_df = pd.read_csv('./dataset/melb_data.csv') # load file from csv .extension to be data frame
housing_df.head() # show the first 5 rows of data
Out[2]:
Identify the Shape of the Dataset¶
In [3]:
housing_df.shape # show the dimension of the dataset with row and column
Out[3]:
Get the List of Columns¶
In [4]:
housing_df.columns # list of columns
Out[4]:
Identify Data Types For Each Column¶
In [5]:
housing_df.dtypes # datatype for every column
Out[5]:
Get Basic Dataset Information¶
In [6]:
housing_df.info() # information of the dataset
Identify Missing Values¶
In [7]:
housing_df.isna().values.any() # detect is there any NULL value in the dataset
Out[7]:
Identify Duplicate Entries/Rows¶
In [8]:
housing_df[housing_df.duplicated(keep=False)] # show all rows with duplication
Out[8]:
In [9]:
housing_df.duplicated().value_counts() # count the total of duplicated data
Out[9]:
Describe the Dataset (summary)¶
In [10]:
housing_df.describe() # data description
Out[10]:
Correlation Matrix¶
In [11]:
housing_df.corr() # correlation between column
Out[11]:
Data Visualization¶
Import Modules¶
In [12]:
import matplotlib.pyplot as plt # import maplotlib as plt for data visualization
import seaborn as sns # import seaborn as sns for data visualization
%matplotlib inline
# output from data visualization data will be directed to notebook
Heatmap¶
In [13]:
sns.heatmap(data=housing_df.corr()) # visualization using Heatmap
Out[13]:
Bar Plot¶
In [14]:
housing_df['Rooms'].value_counts() # count every Suburb
Out[14]:
In [15]:
sns.countplot(data=housing_df, x='Rooms') # visualization using Bar plot with seaborn (colorized)
plt.tight_layout()
Pie Chart¶
In [16]:
housing_df['Rooms'].value_counts().plot.pie(autopct='%1.1f%%', labels=None, legend=True) # visualization using pie chart using percentage
plt.tight_layout()
Scatter Plot¶
In [17]:
sns.scatterplot(x='Bedroom2', y='Car', data=housing_df, hue='Rooms') # visualization using Scatter Plot
plt.tight_layout()
Case: show the widest landsize¶
In [18]:
housing_df.describe().loc['max', 'Landsize'] # find in row first then column
Out[18]:
Data Cleaning¶
In [19]:
housing_df = housing_df.dropna() # drop NA / NULL
In [20]:
housing_df.shape
Out[20]:
Choosing Prediction Target¶
In [21]:
y = housing_df['Price'] # choose Price feature as prediction target then show it
y
Out[21]:
In [22]:
y.describe()
Out[22]:
Choosing Features Selection¶
In [23]:
features = ['Rooms', 'Bathroom', 'Landsize', 'Lattitude', 'Longtitude'] # feature that we use to predict target
X = housing_df[features] # is Rooms, Bathroom, Landsize, Lattitude and Longitude
X
Out[23]:
In [24]:
X.describe() # describing features
Out[24]:
Model Building¶
DecisionTreeRegressor¶
In [25]:
from sklearn.tree import DecisionTreeRegressor # import Decision Tree Regressor as Regressor
Model Configuration¶
In [26]:
housing_model = DecisionTreeRegressor(random_state=1)
Training Model¶
In [27]:
housing_model.fit(X, y)
Out[27]:
Prediction¶
In [28]:
housing_model.predict(X.head()) # predict and show top 5 prediction based on features
Out[28]:
In [29]:
y.head() # compare with original value
Out[29]:
Model Evaluation¶
Mean Absolute Error¶
In [30]:
from sklearn.metrics import mean_absolute_error # Evaluation Metric with MAE
In [31]:
y_hat = housing_model.predict(X) # predict house price from some features inside X variable
mean_absolute_error(y, y_hat) # compare original value with prediction value
Out[31]:
Training module¶
In [32]:
from sklearn.model_selection import train_test_split # as splitter dataset into training and testing set
Split Dataset Into Training and Testing Set¶
In [33]:
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 1)
Config and Train Model¶
In [34]:
housing_model = DecisionTreeRegressor(random_state=1)
housing_model.fit(X_train, y_train)
Out[34]:
Model Evaluation¶
In [35]:
y_hat = housing_model.predict(X_test) # predict in X_test variable
mean_absolute_error(y_test, y_hat) # evaluate y_test with y_hat
Out[35]:
Model optimization¶
Function for Evaluation¶
In [36]:
def get_mae(max_leaf_nodes, X_train, X_test, y_train, y_test):
model = DecisionTreeRegressor(max_leaf_nodes=max_leaf_nodes, random_state=0)
model.fit(X_train, y_train)
y_hat = model.predict(X_test)
mae = mean_absolute_error(y_test, y_hat)
return mae
Create model with Decision Tree with Different parameter/node (max_leaf_nodes)¶
Comparing MAE value with set parameter to choose the the best one among the parameters¶
In [37]:
for max_leaf_nodes in [5, 50, 500, 5000]:
leaf_mae = get_mae(max_leaf_nodes, X_train, X_test, y_train, y_test)
print(f'Max leaf nodes: {max_leaf_nodes} \t Mean Absolute Error: {int(leaf_mae)}')
Random Forest Regressor Exploration¶
Importing RandomForestRegressor¶
In [38]:
from sklearn.ensemble import RandomForestRegressor # import Random Forest Regressor as Regressor
In [39]:
rf_model = RandomForestRegressor(n_estimators=100, random_state=1) # n_estimators=100 means using 100 Decision Tree
rf_model.fit(X_train, y_train) # train data
y_hat = rf_model.predict(X_test) # predict data
print(f'Mean Absolute Error: {int(mean_absolute_error(y_test, y_hat))}') # MAE betweeen y_hat & y_test
