pandas: DataFrame Delete duplicate rows


 

1. Build a DataFrame

 C=pd.DataFrame({'a':['dog']*3+['fish']*3+['dog'],'b':[10,10,12,12,14,14,10]})

 

2. Determine whether there are duplicate items

use duplicated（ ） Function judgment  

 C.duplicated()

 

3.  There are duplicate items , You can use the drop_duplicates() Remove duplicate

 C.drop_duplicates()

 

4. Duplicated（ ） and drop_duplicates（ ） The method is to determine all columns by default （ In the example above, we look at two variables a and b Are they all repeated ）.

We can also judge the repetition of a specific column .

 C.duplicated(['a'])      C.drop_duplicates(['a'])

 C.duplicated(['b'])      C.drop_duplicates(['b'])

 

 

5.  norepeat_df = df.drop_duplicates(subset=['A_ID', 'B_ID'], keep='first')

# Remove the order above UNIT_ID and KPI_ID Duplicate rows in column , And keep the first occurrence of the repeated rows

supplement ： 
When keep=False Time , That is to remove all duplicate lines  
When keep=‘first’ Time , Is to keep the first occurrence of the duplicate line  
When keep=’last’ The last occurrence of the duplicate line is retained . 
（ be careful , The parameter here is a string , Use quotation marks !!!）

 

 

 

 

Technology
©2020-2025 ioDraw All rights reserved, Privacy Policy