首页  思维导图  详情

过程整理

2021-06-11 20:10:44   0  举报





仅支持查看

AI智能生成

过程整理

IOT

作者其他创作

大纲/内容

EDA

pandas_profiling

##依赖库import seaborn as snsimport pandas as pdimport pandas_profiling as ppfimport matplotlib.pyplot as plt

##查看报表report = ppf.ProfileReport(data)report

train.head()#默认显示前五行

full.info()

箱线图

plt.figure(figsize=(10,8))##指定figure的宽和高，单位为英寸sns.boxplot(train.YearBuilt, train.SalePrice)##箱型图是看异常值的，离群点

散点图

plt.figure(figsize=(12,6))plt.scatter(x=train.GrLivArea, y=train.SalePrice)##可以用来观察存在线型的关系plt.xlabel("GrLivArea", fontsize=13)plt.ylabel("SalePrice", fontsize=13)plt.ylim(0,800000)

数据清洗

缺失值

##查看缺失值，并且缺失的个数要从高到低排序miss = full.isnull().sum()#统计出空值的个数miss[miss>0].sort_values(ascending=True)#由低到高排好序full.info()#看空值数量

空值的填充与删除

对字符类型的进行填充

cols1 = ["PoolQC" , "MiscFeature", "Alley", "Fence", "FireplaceQu", "GarageQual", "GarageCond", "GarageFinish", "GarageYrBlt", "GarageType", "BsmtExposure", "BsmtCond", "BsmtQual", "BsmtFinType2", "BsmtFinType1", "MasVnrType"]for col in cols1: full[col].fillna("None",inplace=True)

对数值类型的进行填充

cols=["MasVnrArea", "BsmtUnfSF", "TotalBsmtSF", "GarageCars", "BsmtFinSF2", "BsmtFinSF1", "GarageArea"]for col in cols: full[col].fillna(0, inplace=True)

对某一列空值进行填充（用这一列的均值）

full["LotFrontage"].fillna(np.mean(full["LotFrontage"]),inplace=True)

对这些列进行众数填充

cols2 = ["MSZoning", "BsmtFullBath", "BsmtHalfBath", "Utilities", "Functional", "Electrical", "KitchenQual", "SaleType","Exterior1st", "Exterior2nd"]for col in cols2: full[col].fillna(full[col].mode()[0], inplace=True)

查看哪些是还没填充好的

full.isnull().sum()[full.isnull().sum()>0]##至此我们已经把空值填充好了

离群点处理

##pandas 里面的条件索引train.drop(train[(train["GrLivArea"]>4000)&(train["SalePrice"]<300000)].index,inplace=True)

合并数据

pd.concat

full = pd.concat([train,test],ignore_index=True)

去除id列

full.drop("Id",axis=1,inplace=True)

对不连续的数字或者文本进行数字化处理

sklearn.preprocessing.LableEncoder

for col in cols2: full[col]=full[col].astype(str)##astype来进行数据转换成字符串类型