Data Cleaning in Pandas | Python Pandas Tutorials

For those struggling with the regular expression at 14:57 , you might need to explicitly assign regex = True (based on the FutureWarning displayed in the video). That is: df['Phone_Number'] = df['Phone_Number'].str.replace('[^a-zA-Z0-9]', '', regex=True)

Fan from India I just got 2 offers from very good companies thanks to your videos and it helped me transition from a customer success support to Data Analyst

For splitting the address at 21:29, you may want to add a named parameter to the value of 2, as in n=2: df[["Street_Address", "State", "Zip_Code"]] = df["Address"].str.split(',', n=2, expand=True)

Some of the phone numbers are removed while doing the formatting. If you look in the excel file, you'll see that some of the numbers are strings and some are integers. When you run the string method during the formatting, it replaces the numeric values with NaN and they are later removed completely. If you want to avoid losing that data you'll need to use df["Phone_Number"] = df["Phone_Number"].astype(str) before formatting. You also won't need to convert to string in the lambda after doing this.

Thanks for this content, this was so helpful!! I think i have some optimizations, correct me if im wrong :D 27:04 instead of calling the replace function multiple times, you can create a mapping just like: replace_mapping = {'Yes': 'Y', 'No': 'N'} and call it like: df = df.replace(replace_mapping), so you dont have to specify mapping for each column and need to call .replace() just once. 34:16 instead of the for loop + manually dropping row per row, you can make use of the .loc function like: df = df.loc[df["Do_Not_Contact"] == "N"] in order to filter the rows based on filter criterium.

I am studying Data Collection and Data Visualization at Kings College, your channel is reccomned by our lecturers to understand data cleaning.

I like how in some of your videos you show us the long way and then the short cut, instead of just showing the short cut. I think that way gives the person who is learning a better breakdown of what they are doing.

14:45 run: df['Phone_Number'].str.replace('[^a-zA-Z0-9]','', regex = True) for newer versions of Anaconda. The code df['Phone_Number'].str.replace('[^a-zA-Z0-9]','') has a potential issue. The regular expression [^a-zA-Z0-9] is correct for matching any non-alphanumeric character, but the .str.replace() method now requires the regex=True parameter to be explicitly set when using regex patterns like this. Without regex=True, pandas will treat the pattern as a literal string, which won't work as intended. 🤝

Also, to clean the Do_Not_Contact field, one can use: df['Do_Not_Contact'] = df['Do_Not_Contact'].replace({'N': 'No', 'Y': 'Yes'})

what seems to be a daunting task at the beginning turns out to have an easy explanation with the right tools, thank you Alex !!!