Using Pandas in Python
Hello all,
This is the fourth article in the series Python for Data Science. If you are new to this series, we would recommend you to read our previous articles
To install pandas in your python environment, please use the below command
pip install pandas
Please refer the videos below for detailed explanation on Pandas
After you have installed Pandas, please refer the following notebook to understand on how to use Pandas functionalities.
In [1]:
import pandas as pd
import os
Creating a DataFrame¶
In [2]:
data1 = [1,2,3]
data2 = [4,5,6]
data3 = [7,8,9]
pd.DataFrame([data1,data2,data3],columns = ['a','b','c'],index = ['row1','row2','row3'])
Out[2]:
In [3]:
pd.DataFrame([data1,data2,data3])
Out[3]:
Reading CSV thru Pandas¶
In [4]:
csv_file_path = r"E:\openknowledgeshare.blogspot.com\Python\Data\SBIN.NS.csv"
In [5]:
df = pd.read_csv(csv_file_path)
df
Out[5]:
In [6]:
df.shape
Out[6]:
In [7]:
df.columns
Out[7]:
In [8]:
df.rename(columns={'Date':'NewDate'},inplace=True)
In [9]:
df
Out[9]:
In [10]:
df.head()
Out[10]:
In [11]:
df.head(10)
Out[11]:
In [12]:
df.tail()
Out[12]:
In [13]:
df.describe()
Out[13]:
In [14]:
df.head()
Out[14]:
In [15]:
df.set_index('NewDate').head()
Out[15]:
In [16]:
df["High"].head()
Out[16]:
In [17]:
type(df)
Out[17]:
In [18]:
type(df["High"])
Out[18]:
In [19]:
df["High"].to_frame().head()
Out[19]:
In [20]:
df.iloc[10:20]
Out[20]:
In [21]:
df.iloc[10:20][["Close","High"]]
Out[21]:
In [22]:
df.iloc[[1,3,10]][["Close","High"]]
Out[22]:
In [23]:
a = df.head().values
a
Out[23]:
In [24]:
a.shape
Out[24]:
In [25]:
type(a)
Out[25]:
In [26]:
df.head()
Out[26]:
In [ ]:
Manipulating Cell Values of Dataframe¶
In [27]:
data1 = [1,2,3]
data2 = [4,5,6]
data3 = [7,8,9]
df1 = pd.DataFrame([data1,data2,data3],columns = ['a','b','c'])
df1
Out[27]:
In [28]:
df1.iloc[1]['c']
Out[28]:
In [29]:
df1.iloc[1]['c'] = 10
df1
Out[29]:
In [30]:
df1.iloc[2]
Out[30]:
In [31]:
type(df1.iloc[2])
Out[31]:
In [32]:
df1.iloc[2] = [10,20,30]
In [33]:
df1
Out[33]:
In [34]:
df1.iloc[[1,2]] = [[100,200,300],[1030,2010,3000]]
In [35]:
df1
Out[35]:
In [36]:
df1.iloc[3] = [10,4,1]
Appending Row to DataFrame¶
In [37]:
a = [1,2,3]
a.append(5)
a
Out[37]:
In [38]:
df1
Out[38]:
In [39]:
df1.append([[10,8,4]])
Out[39]:
In [40]:
df1
Out[40]:
Case 1: DataFrame without Column Names¶
In [41]:
data1 = [1,2,3]
data2 = [4,5,6]
data3 = [7,8,9]
df2 = pd.DataFrame([data1,data2,data3])
df2
Out[41]:
In [42]:
df2 = df2.append([[10,20,30]])
df2
Out[42]:
In [43]:
df2.reset_index()
Out[43]:
In [44]:
df2.reset_index(drop=True,inplace = True)
df2
Out[44]:
Case2 : DataFrame with Column Names¶
In [45]:
df1
Out[45]:
In [46]:
df1.append([[10,20,30]],ignore_index=True)
Out[46]:
In [47]:
df1.append({'a':10,'b':20,'c':30},ignore_index=True)
Out[47]:
In [48]:
columnnames_list = list(df1.columns)
columnnames_list
Out[48]:
In [49]:
data = [1000,2000,3000]
In [50]:
data_dict = {}
for i in range(0,len(columnnames_list)):
each_column = columnnames_list[i]
this_data_point = data[i]
data_dict[each_column] = this_data_point
data_dict
Out[50]:
In [51]:
df1 = df1.append(data_dict,ignore_index = True)
df1
Out[51]:
In [52]:
df1
Out[52]:
Sampling DataFrame¶
In [53]:
import numpy as np
In [54]:
a = [1,2,3,5,6,7]
np.random.choice(a,3)
Out[54]:
In [55]:
df.sample(10)
Out[55]:
Merge, Join and Concatenate DataFrames¶
In [56]:
df1
Out[56]:
In [57]:
df2
Out[57]:
In [58]:
df3 = pd.merge(df1,df2,left_index=True,right_index=True,how='outer')
df3
Out[58]:
In [59]:
data1 = [1,2,3]
data2 = [4,5,6]
data3 = [7,8,9]
df2 = pd.DataFrame([data1,data2,data3])
df2
Out[59]:
In [60]:
df3 = pd.merge(df1,df2,left_index=True,right_index=True,how='outer')
df3
Out[60]:
In [61]:
df3 = pd.merge(df1,df2,left_index=True,right_index=True,how='inner')
df3
Out[61]:
In [62]:
df3 = pd.merge(df1,df2,left_index=True,right_index=True,how='left')
df3
Out[62]:
In [63]:
df3 = pd.merge(df1,df2,left_index=True,right_index=True,how='right')
df3
Out[63]:
In [64]:
df1 = pd.DataFrame({'key': ['K0', 'K1', 'K2', 'K3'],
'A': ['A0', 'A1', 'A2', 'A3'],
'B': ['B0', 'B1', 'B2', 'B3']})
df2 = pd.DataFrame({'key': ['K0', 'K1', 'K4', 'K3'],
'C': ['C0', 'C1', 'C4', 'C3'],
'D': ['D0', 'D1', 'D4', 'D3']})
In [65]:
df1
Out[65]:
In [66]:
df2
Out[66]:
In [67]:
df3 = pd.merge(df1,df2,left_index=True,right_index=True,how='outer')
df3
Out[67]:
In [68]:
df3 = pd.merge(df1,df2,on ='key',how='outer')
df3
Out[68]:
In [69]:
df3 = pd.merge(df1,df2,on ='key',how='inner')
df3
Out[69]:
In [70]:
df3 = pd.merge(df1,df2,on ='key',how='left')
df3
Out[70]:
In [71]:
df3 = pd.merge(df1,df2,on ='key',how='right')
df3
Out[71]:
Concat¶
In [72]:
s3 = pd.Series([0, 1, 2, 3], name='foo')
s4 = pd.Series([4, 5, 6, 7,12])
s5 = pd.Series([11, 10, 9, 8,13,14])
In [73]:
s3
Out[73]:
In [74]:
s4
Out[74]:
In [75]:
s5
Out[75]:
In [76]:
pd.concat([s3, s4, s5], axis=1)
Out[76]:
In [77]:
pd.concat([s3, s4, s5], axis=0)
Out[77]:
In [78]:
df1
Out[78]:
In [79]:
df2
Out[79]:
In [80]:
df1.append(df2)
Out[80]:
In [81]:
df3 = pd.concat([df1,df2], ignore_index=True, sort=False)
df3
Out[81]:
Changing Column Order¶
In [82]:
df3[['A','B','C','D','key']]
Out[82]:
Handling Missing Values¶
In [83]:
df3
Out[83]:
In [84]:
df4 = df.head().copy()
df4
Out[84]:
In [85]:
df3
Out[85]:
In [86]:
df4
Out[86]:
In [87]:
df5 = pd.concat([df3,df4],ignore_index=True, sort=False)
df5
Out[87]:
In [88]:
df5.isna()
Out[88]:
In [89]:
df5.fillna("EMPTY")
Out[89]:
In [90]:
df5.replace(np.nan,0,inplace = True)
In [91]:
df5
Out[91]:
Filtering DataFrame¶
In [92]:
df5['key']
Out[92]:
In [93]:
df5[df5['key'] == 'K0']
Out[93]:
In [94]:
df5[(df5['key'] == 'K4') & (df5['Open'] == 0)]
Out[94]:
In [95]:
keylist = ['K2','K3']
df5[(df5['key'].isin(keylist)) & (df5['Open'] == 0)]
Out[95]:
In [96]:
df5[df5['Open'] < 100]
Out[96]:
Renaming Columns¶
In [97]:
temp = df.sample(10).copy()
temp
Out[97]:
In [98]:
temp.rename(columns = {'NewDate':'OneMoreNewDate','Adj Close':'ABC'},inplace = True)
temp
Out[98]:
In [99]:
temp.columns = ['A','B','CD','EF','GH','i',3939]
temp
Out[99]:
Deleting Column¶
In [100]:
temp.drop("B",axis=1,inplace = True)
temp
Out[100]:
In [101]:
temp.drop(724,axis=0,inplace = True)
temp
In [102]:
temp.drop([1499,1145],axis=0,inplace = True)
temp
In [103]:
temp.drop("GH",axis=1,inplace = True)
temp
Out[103]:
In [104]:
df1 = pd.DataFrame({'key': ['K0', 'K1', 'K2', 'K3'],
'A': ['A0', 'A1', 'A2', 'A3'],
'B': ['B0', 'B1', 'B2', 'B3']})
df2 = pd.DataFrame({'key': ['K0', 'K1', 'K4', 'K3'],
'C': ['C0', 'C1', 'C4', 'C3'],
'D': ['D0', 'D1', 'D4', 'D3']})
In [105]:
df1
Out[105]:
In [106]:
df1.drop('A',axis=1,inplace = True)
df1
Out[106]:
In [107]:
df1.drop(2,axis=0,inplace = True)
df1
Out[107]:
Rowwise and Column wise Fuctions¶
In [108]:
temp
Out[108]:
In [109]:
temp['CD'].dtype
Out[109]:
In [110]:
temp['CD'] = temp['CD'].astype('int32')
temp['CD']
In [111]:
temp
Out[111]:
In [112]:
def range_indicator(input_value):
output_value = False
if(input_value > 50):
output_value = True
return output_value
In [113]:
range_indicator(278)
Out[113]:
In [114]:
temp['CD']
Out[114]:
In [115]:
temp['Range_Indicator'] = temp['CD'].apply(lambda x: range_indicator(x))
temp
Out[115]:
In [116]:
lambda x: range_indicator(x)
Out[116]:
In [117]:
for each_item in [12,4,5,321]:
print(range_indicator(each_item))
In [118]:
list(map(range_indicator,[12,4,5,321]))
Out[118]:
In [119]:
list(map(range_indicator,temp['CD']))
Out[119]:
In [ ]:
Pandas Visualization¶
In [120]:
import matplotlib.pyplot as plt
In [121]:
df.head()
Out[121]:
In [122]:
df.tail()
Out[122]:
In [123]:
df.plot(x='NewDate',y = 'Open')
Out[123]:
In [124]:
df.plot(x='Open',y = 'Close',kind='scatter')
Out[124]:
In [125]:
df['Open'].hist()
Out[125]:
In [126]:
df.shape
Out[126]:
Statistics¶
In [127]:
df.head()
Out[127]:
In [130]:
df.info()
In [131]:
df.describe()
Out[131]:
In [132]:
df["Open"].hist()
Out[132]:
In [194]:
df.corr()
Out[194]:
In [200]:
df.head()
Out[200]:
In [204]:
df.head().sum(axis=0).sum()
Out[204]:
In [133]:
a = [1,2,3,4,5,10,20,50]
In [134]:
sum(a)/len(a)
Out[134]:
In [136]:
#Standard Deviation
import numpy as np
In [137]:
a = np.asarray(a)
In [138]:
a
Out[138]:
In [139]:
np.mean(a)
Out[139]:
In [140]:
np.std(a)
Out[140]:
In [141]:
df.head()
Out[141]:
In [142]:
df1
Out[142]:
In [146]:
df.apply(lambda x: x['Open']+ x['High'])
Out[146]:
In [149]:
df[["Open","High","Low","Close"]].apply(lambda x: 2*x)
Out[149]:
In [154]:
df["NewDate"] = pd.to_datetime(df["NewDate"],format='%Y-%m-%d')
In [155]:
df["NewDate"].dtype
Out[155]:
In [159]:
df['NewDate'].iloc[0].day
Out[159]:
In [161]:
df['NewDate'].iloc[0].date()
Out[161]:
In [162]:
import datetime
In [166]:
datetime.timedelta(days=10,hours=10)
Out[166]:
In [168]:
df['NewDate'].iloc[0] + datetime.timedelta(days=10,hours=10)
Out[168]:
In [169]:
a = df['NewDate'].iloc[0] + datetime.timedelta(days=10,hours=10)
In [170]:
a
Out[170]:
In [171]:
a.strptime()
In [172]:
import datetime
d = datetime.datetime.strptime("01/27/2012", "%m/%d/%Y")
In [173]:
d
Out[173]:
In [175]:
b = datetime.datetime.strftime(d,"%Y-%m-%d")
b
Out[175]:
In [176]:
type(b)
Out[176]:
In [178]:
import time
In [181]:
time.ctime()
Out[181]:
TimeStamp Folder Creation¶
In [187]:
timestamp_now = datetime.datetime.now().strftime('%Y%m%d_%H%M%S')
In [188]:
import os
In [191]:
timestamp_now = datetime.datetime.now().strftime('%Y%m%d_%H%M%S')
main_folder_path = r'E:\openknowledgeshare.blogspot.com\Python\Outputs'
os.makedirs(os.path.join(main_folder_path,timestamp_now))
Time Taken¶
In [193]:
start_time = time.clock()
for i in range(0,10):
i
end_time = time.clock()
print("Time taken for this code is {} seconds".format(end_time - start_time))
Stacking¶
In [205]:
df1
Out[205]:
In [206]:
df2
Out[206]:
In [207]:
df2.stack()
Out[207]:
In [208]:
df2.unstack('')
Out[208]:
In [211]:
import pandas as pd
import numpy as np
header = pd.MultiIndex.from_product([['Semester1','Semester2'],['Maths','Science']])
d=([[12,45,67,56],[78,89,45,67],[45,67,89,90],[67,44,56,55]])
df = pd.DataFrame(d,
columns=['Alisa','Bobby','Cathrine','Jack'],
index=header)
In [212]:
df
Out[212]:
In [214]:
df.xs("Semester1")
Out[214]:
In [215]:
df.xs("Semester2")
Out[215]:
In [219]:
df.xs("Maths",level=1)
Out[219]:
In [220]:
df
Out[220]:
In [224]:
df.unstack()
Out[224]:
In [ ]:
Comments
Post a Comment