Statistical Language Model
STATISTICAL LANGUAGE MODELING
• STATISTICAL LANGUAGE MODELING IS THE DEVELOPMENT OF PROBABILISTIC MODELS THAT CAN PREDICT THE
NEXT WORD IN THE SEQUENCE GIVEN THE WORDS THAT PRECEDE IT.
• A STATISTICAL LANGUAGE MODEL LEARNS THE PROBABILITY OF WORD OCCURRENCE BASED ON EXAMPLES OF
TEXT.
• SIMPLER MODELS MAY LOOK AT A CONTEXT OF A SHORT SEQUENCE OF WORDS, WHEREAS LARGER MODELS MAY
WORK AT THE LEVEL OF SENTENCES OR PARAGRAPHS. MOST COMMONLY, LANGUAGE MODELS OPERATE AT THE
LEVEL OF WORDS.
TYPES OF STATISTICAL LANGUAGE
MODELS
• 1. N-GRAM
THIS IS ONE OF THE SIMPLEST APPROACHES TO LANGUAGE MODELLING. HERE, A PROBABILITY DISTRIBUTION FOR A
SEQUENCE OF ‘N’ IS CREATED, WHERE ‘N’ CAN BE ANY NUMBER AND DEFINES THE SIZE OF THE GRAM (OR SEQUENCE
OF WORDS BEING ASSIGNED A PROBABILITY). IF N=4, A GRAM MAY LOOK LIKE: “CAN YOU HELP ME”. BASICALLY, ‘N’ IS
THE AMOUNT OF CONTEXT THAT THE MODEL IS TRAINED TO CONSIDER. THERE ARE DIFFERENT TYPES OF N-GRAM
MODELS, SUCH AS UNIGRAMS, BIGRAMS, TRIGRAMS, ETC.
2. EXPONENTIAL
• THIS TYPE OF STATISTICAL MODEL EVALUATES TEXT BY USING AN EQUATION
THAT IS A COMBINATION OF N-GRAMS AND FEATURE FUNCTIONS. HERE, THE
FEATURES AND PARAMETERS OF THE DESIRED RESULTS ARE ALREADY
SPECIFIED. THE MODEL IS BASED ON THE PRINCIPLE OF ENTROPY, WHICH
STATES THAT A PROBABILITY DISTRIBUTION WITH THE MOST ENTROPY IS THE
BEST CHOICE. EXPONENTIAL MODELS HAVE FEWER STATISTICAL
ASSUMPTIONS, WHICH MEANS THE CHANCES OF HAVING ACCURATE RESULTS
ARE HIGHER.
3. CONTINUOUS SPACE
• IN THIS TYPE OF STATISTICAL MODEL, WORDS ARE ARRANGED AS A NON-LINEAR
COMBINATION OF WEIGHTS IN A NEURAL NETWORK. THE PROCESS OF ASSIGNING
WEIGHT TO A WORD IS KNOWN AS WORD EMBEDDING. THIS TYPE OF MODEL
PROVES HELPFUL IN SCENARIOS WHERE THE DATA SET OF WORDS CONTINUES TO
BECOME LARGE AND INCLUDES UNIQUE WORDS.
• IN CASES WHERE THE DATA SET IS LARGE AND CONSISTS OF RARELY USED OR
UNIQUE WORDS, LINEAR MODELS SUCH AS N-GRAMS DO NOT WORK. THIS IS
BECAUSE, WITH INCREASING WORDS, THE POSSIBLE WORD SEQUENCES INCREASE,
AND THUS THE PATTERNS PREDICTING THE NEXT WORD BECOME WEAKER.
BUILD A SIMPLE STATISTICAL LANGUAGE MODEL
• LANGUAGE MODELS START WITH A MARKOV ASSUMPTION. THIS IS A SIMPLIFYING
ASSUMPTION THAT THE K+1ST WORD IS DEPENDENT ON THE PREVIOUS K WORDS. A
2ND ORDER ASSUMPTION RESULTS IN A BIGRAM MODEL. THE MODELS ARE TRAINED
USING MAXIMUM LIKELIHOOD ESTIMATION (MLE) OF AN EXISTING CORPUS. THE MLE
APPROACH, THEN, IS SIMPLY A FRACTION OF WORK COUNTS.
THERE ARE SOME ADVANTAGES TO USING TRADITIONAL N-GRAM LANGUAGE MODELS.
• THEY ARE EASY TO TRAIN ON A LARGE CORPUS
• THEY WORK SURPRISINGLY WELL IN MOST TASKS!!
• HOWEVER, THEY HAVE SOME DISADVANTAGES
APPLICATION OF STATISTICAL LANGUAGE
SPEECH RECOGNITION
• VOICE ASSISTANTS SUCH AS SIRI AND ALEXA ARE EXAMPLES OF HOW LANGUAGE MODELS
HELP MACHINES IN PROCESSING SPEECH AUDIO.
MACHINE TRANSLATION
• GOOGLE TRANSLATE AND MICROSOFT TRANSLATE ARE EXAMPLES OF HOW NLP MODELS
CAN HELP IN TRANSLATING ONE LANGUAGE TO ANOTHER.
CONT.……
SENTIMENT ANALYSIS
• THIS HELPS IN ANALYZING SENTIMENTS BEHIND A PHRASE. THIS USE CASE OF NLP MODELS IS USED IN
PRODUCTS THAT ALLOW BUSINESSES TO UNDERSTAND A CUSTOMER’S INTENT BEHIND OPINIONS OR
ATTITUDES EXPRESSED IN THE TEXT. HUBSPOT’S SERVICE HUB IS AN EXAMPLE OF HOW LANGUAGE MODELS
CAN HELP IN SENTIMENT ANALYSIS.
TEXT SUGGESTIONS
• GOOGLE SERVICES SUCH AS GMAIL OR GOOGLE DOCS USE LANGUAGE MODELS TO HELP USERS GET TEXT
SUGGESTIONS WHILE THEY COMPOSE AN EMAIL OR CREATE LONG TEXT DOCUMENTS, RESPECTIVELY.
PARSING TOOLS
• PARSING INVOLVES ANALYZING SENTENCES OR WORDS THAT COMPLY WITH SYNTAX OR GRAMMAR RULES.
SPELL-CHECKING TOOLS ARE PERFECT EXAMPLES OF LANGUAGE MODELLING AND PARSING.