Costa Rica Foreigners ID Numbers¶
Introduction¶
The function clean_cr_cr()
cleans a column containing Costa Rica foreigners ID number (CR) strings, and standardizes them in a given format. The function validate_cr_cr()
validates either a single CR strings, a column of CR strings or a DataFrame of CR strings, returning True
if the value is valid, and False
otherwise.
CR strings can be converted to the following formats via the output_format
parameter:
compact
: only number strings without any seperators or whitespace, like “122200569906”standard
: CR strings with proper whitespace in the proper places. Note that in the case of CR, the compact format is the same as the standard one.
Invalid parsing is handled with the errors
parameter:
coerce
(default): invalid parsing will be set to NaNignore
: invalid parsing will return the inputraise
: invalid parsing will raise an exception
The following sections demonstrate the functionality of clean_cr_cr()
and validate_cr_cr()
.
An example dataset containing CR strings¶
[1]:
import pandas as pd
import numpy as np
df = pd.DataFrame(
{
"cr": [
'122200569906',
'12345678',
'BE 428759497',
'BE431150351',
"002 724 334",
"hello",
np.nan,
"NULL",
],
"address": [
"123 Pine Ave.",
"main st",
"1234 west main heights 57033",
"apt 1 789 s maple rd manhattan",
"robie house, 789 north main street",
"1111 S Figueroa St, Los Angeles, CA 90015",
"(staples center) 1111 S Figueroa St, Los Angeles",
"hello",
]
}
)
df
[1]:
cr | address | |
---|---|---|
0 | 122200569906 | 123 Pine Ave. |
1 | 12345678 | main st |
2 | BE 428759497 | 1234 west main heights 57033 |
3 | BE431150351 | apt 1 789 s maple rd manhattan |
4 | 002 724 334 | robie house, 789 north main street |
5 | hello | 1111 S Figueroa St, Los Angeles, CA 90015 |
6 | NaN | (staples center) 1111 S Figueroa St, Los Angeles |
7 | NULL | hello |
1. Default clean_cr_cr
¶
By default, clean_cr_cr
will clean cr strings and output them in the standard format with proper separators.
[2]:
from dataprep.clean import clean_cr_cr
clean_cr_cr(df, column = "cr")
[2]:
cr | address | cr_clean | |
---|---|---|---|
0 | 122200569906 | 123 Pine Ave. | 122200569906 |
1 | 12345678 | main st | NaN |
2 | BE 428759497 | 1234 west main heights 57033 | NaN |
3 | BE431150351 | apt 1 789 s maple rd manhattan | NaN |
4 | 002 724 334 | robie house, 789 north main street | NaN |
5 | hello | 1111 S Figueroa St, Los Angeles, CA 90015 | NaN |
6 | NaN | (staples center) 1111 S Figueroa St, Los Angeles | NaN |
7 | NULL | hello | NaN |
2. Output formats¶
This section demonstrates the output parameter.
standard
(default)¶
[3]:
clean_cr_cr(df, column = "cr", output_format="standard")
[3]:
cr | address | cr_clean | |
---|---|---|---|
0 | 122200569906 | 123 Pine Ave. | 122200569906 |
1 | 12345678 | main st | NaN |
2 | BE 428759497 | 1234 west main heights 57033 | NaN |
3 | BE431150351 | apt 1 789 s maple rd manhattan | NaN |
4 | 002 724 334 | robie house, 789 north main street | NaN |
5 | hello | 1111 S Figueroa St, Los Angeles, CA 90015 | NaN |
6 | NaN | (staples center) 1111 S Figueroa St, Los Angeles | NaN |
7 | NULL | hello | NaN |
compact
¶
[4]:
clean_cr_cr(df, column = "cr", output_format="compact")
[4]:
cr | address | cr_clean | |
---|---|---|---|
0 | 122200569906 | 123 Pine Ave. | 122200569906 |
1 | 12345678 | main st | NaN |
2 | BE 428759497 | 1234 west main heights 57033 | NaN |
3 | BE431150351 | apt 1 789 s maple rd manhattan | NaN |
4 | 002 724 334 | robie house, 789 north main street | NaN |
5 | hello | 1111 S Figueroa St, Los Angeles, CA 90015 | NaN |
6 | NaN | (staples center) 1111 S Figueroa St, Los Angeles | NaN |
7 | NULL | hello | NaN |
3. inplace
parameter¶
This deletes the given column from the returned DataFrame. A new column containing cleaned CR strings is added with a title in the format "{original title}_clean"
.
[5]:
clean_cr_cr(df, column="cr", inplace=True)
[5]:
cr_clean | address | |
---|---|---|
0 | 122200569906 | 123 Pine Ave. |
1 | NaN | main st |
2 | NaN | 1234 west main heights 57033 |
3 | NaN | apt 1 789 s maple rd manhattan |
4 | NaN | robie house, 789 north main street |
5 | NaN | 1111 S Figueroa St, Los Angeles, CA 90015 |
6 | NaN | (staples center) 1111 S Figueroa St, Los Angeles |
7 | NaN | hello |
4. errors
parameter¶
coerce
(default)¶
[6]:
clean_cr_cr(df, "cr", errors="coerce")
[6]:
cr | address | cr_clean | |
---|---|---|---|
0 | 122200569906 | 123 Pine Ave. | 122200569906 |
1 | 12345678 | main st | NaN |
2 | BE 428759497 | 1234 west main heights 57033 | NaN |
3 | BE431150351 | apt 1 789 s maple rd manhattan | NaN |
4 | 002 724 334 | robie house, 789 north main street | NaN |
5 | hello | 1111 S Figueroa St, Los Angeles, CA 90015 | NaN |
6 | NaN | (staples center) 1111 S Figueroa St, Los Angeles | NaN |
7 | NULL | hello | NaN |
ignore
¶
[7]:
clean_cr_cr(df, "cr", errors="ignore")
[7]:
cr | address | cr_clean | |
---|---|---|---|
0 | 122200569906 | 123 Pine Ave. | 122200569906 |
1 | 12345678 | main st | 12345678 |
2 | BE 428759497 | 1234 west main heights 57033 | BE 428759497 |
3 | BE431150351 | apt 1 789 s maple rd manhattan | BE431150351 |
4 | 002 724 334 | robie house, 789 north main street | 002 724 334 |
5 | hello | 1111 S Figueroa St, Los Angeles, CA 90015 | hello |
6 | NaN | (staples center) 1111 S Figueroa St, Los Angeles | NaN |
7 | NULL | hello | NaN |
4. validate_cr_cr()
¶
validate_cr_cr()
returns True
when the input is a valid CR. Otherwise it returns False
.
The input of validate_cr_cr()
can be a string, a Pandas DataSeries, a Dask DataSeries, a Pandas DataFrame and a dask DataFrame.
When the input is a string, a Pandas DataSeries or a Dask DataSeries, user doesn’t need to specify a column name to be validated.
When the input is a Pandas DataFrame or a dask DataFrame, user can both specify or not specify a column name to be validated. If user specify the column name, validate_cr_cr()
only returns the validation result for the specified column. If user doesn’t specify the column name, validate_cr_cr()
returns the validation result for the whole DataFrame.
[8]:
from dataprep.clean import validate_cr_cr
print(validate_cr_cr("122200569906"))
print(validate_cr_cr("12345678"))
print(validate_cr_cr('BE 428759497'))
print(validate_cr_cr('BE431150351'))
print(validate_cr_cr("004085616"))
print(validate_cr_cr("hello"))
print(validate_cr_cr(np.nan))
print(validate_cr_cr("NULL"))
True
False
False
False
False
False
False
False
Series¶
[9]:
validate_cr_cr(df["cr"])
[9]:
0 True
1 False
2 False
3 False
4 False
5 False
6 False
7 False
Name: cr, dtype: bool
DataFrame + Specify Column¶
[10]:
validate_cr_cr(df, column="cr")
[10]:
0 True
1 False
2 False
3 False
4 False
5 False
6 False
7 False
Name: cr, dtype: bool
Only DataFrame¶
[11]:
validate_cr_cr(df)
[11]:
cr | address | |
---|---|---|
0 | True | False |
1 | False | False |
2 | False | False |
3 | False | False |
4 | False | False |
5 | False | False |
6 | False | False |
7 | False | False |
[ ]: