Skip to content

normalize()

The primary normalization API in BASA.

normalize() provides fine-grained control over the entire normalization pipeline, including slang conversion, typo correction, punctuation reduction, and case handling.

Import

from basa import normalize

Basic Usage

normalize(
    "GW GKKKK NGERTIII BNGTTTT!!!!!"
)

Output:

'saya tidak mengerti banget!'

Function Signature

normalize(
    text,
    apply_slang=True,
    apply_typo=False,
    lowercase=True,
    normalize_punctuation=True,
    normalize_whitespace=True,
)

Parameters

text

Type:

str | list[str]

The input text or batch of texts to normalize.

apply_slang

Default:

True

Applies:

  • slang dictionary replacement
  • repeated-character reduction

Example:

normalize(
    "gw gk ngerti",
    apply_slang=False
)

Output:

'gw gk ngerti'

apply_typo

Default:

False

Enables Levenshtein-based typo correction.

A vocabulary must be loaded beforehand:

from basa import typo

typo.add_to_vocab({
    "makan",
    "minum"
})

lowercase

Default:

True

Convert text to lowercase before normalization.

Example:

normalize(
    "Jokowi pergi ke Jakarta",
    lowercase=False
)

Output:

'Jokowi pergi ke Jakarta'

normalize_punctuation

Default:

True

Collapses repeated punctuation:

!!!!! → !
????? → ?
..... → .

normalize_whitespace

Default:

True

Collapses multiple spaces and trims leading/trailing whitespace.

Batch Processing

normalize([
    "gw gk ngerti",
    "udh makan blm?"
])

Output:

[
    'saya tidak mengerti',
    'sudah makan belum?'
]

Design Philosophy

normalize() follows several principles:

  • Conservative defaults
  • Explicit configuration
  • Predictable behavior
  • Stable public APIs
  • Production-friendly preprocessing

For most users, quick() is recommended as the simpler entry point.