PROGRAMMING BASICS

N AT U R A L L A N G U A G E
SOCIAL NET WORKS

J AC K HO NG
2015
AGENDA
• Introduction
• Basic programming logic
• Proc SQL in SAS
• Important procedures in Financial Studies
– Fixed effects and error corrections in SAS
– Instrumented regressions in Stata
– Tobit 2 (selection) and Tobit 5 (treatment) in Stata
– Calendar time portfolio (alpha)
• Special topics
– Natural language processing
– Graphics
– Social networks
 COMPUTING BASICS (HARDWARE)
• What does a computer do?
– A computer computes!

• Using the biological brain as an analogy

Central Processing Unit (CPU) Random Access Memory (RAM) Disk Drives
= = =
Frontal Lobe (calculates) Working (fast) memory Long term (slow) memory
 COMPUTERS VS HUMANS
• Computers beat brains in terms of
– Speed (computation and recall)
• How long do you think it will take my
MacBook Pro to compute (1 + 1), 100
million times?
– Accuracy (perfect recall)
• Will a computer system return different
data the same way you call it every time?
– Working volume
• How many wikipedia (entire) can a
S$140 hard disk hold?
COMPUTERS VS HUMANS
• The common saying is that computers are
way behind humans in higher cognitive skills,
such as
– Relationship mapping
– Synthesizing different ideas
– Extracting contexts
REALLY?
• Context-based AI
– Siri
• “PWN” Jeopardy
– IBM’s Watson
• Predict human behavior
– Telling you what you want before
you know it
– Amazon, Tesco etc
• Deep Learning
– Without teaching a program what
is the definition of a cat, it can
differentiate a cat from other
animals
WE ARE EQUIPPED FOR THE FUTURE
OF DATA SCIENCE
• The core value-add of these futuristic applications lies in the
ability to make sense of what has happened, is happening, and
predict what will happen
• The tools in which this core value-add is built upon
– Insane amounts of data
– Data varieties
– Statistics, econometrics and other quantitative scientific
methods
• Computing resources and skillsets are important because
they are the only way to unlock this core value-add
• The future of data science is not tailored for computer
science specialists, but to applied scientists who are
competent in the art of computer science
WHY SAS WILL NOT BE THE DOMINANT
LANGUAGE/PLATFORM OF DATA SCIENTISTS
• SAS is a proprietary language (database-
driven) and programming competencies in it
are not fully transferable
– Uses syntax and logic that are partially independent
from mainstream programming languages
• Many common logic are not supported by SAS, or
require convoluted syntax and steps
• Users have easy control over the data manipulation
part, but the procedural coding part is complex
(PROC IML) – Ask MATLAB and R users

• SAS’s architecture is rigid

– It is hard to mix and match data technologies
– Compatibility issues are rife
WHY SAS WILL NOT BE THE DOMINANT
LANGUAGE/PLATFORM OF DATA SCIENTISTS
• SAS is slow to update
– R (and Stata) has a wider variety of recent statistical
procedures
• SAS has weak visualization capabilities
• R and other open-source languages have a large supporting
community that improves and extends the language and
modules (anybody can do it)
• SAS cannot handle programming requirements outside its
limited scope
– E.g. Go to [Link], extract the json (javascript
object notation) list of historical exchange rates for hundreds
of countries, munge them into a dataset and perform analysis?
– Natural language? Social networks?
WHY DO WE STILL LEARN ?
• First word: Legacy / Second word: Convenience
• Academia legacy (Finance)
– WRDS datasets are in SAS
– WRDS server (Unix and Windows) uses PC-SAS
– Many empirical researchers in Finance uses SAS, and their SAS codes can
be found in WRDS
• SAS is stable and easier to code for brute-force data manipulation
– Handling large datasets using open-source software typically require more
codes to link up operations between CPU, RAM, HDD and Databases
– SAS is both a database and programming platform, and its proprietary
codes (e.g. PROC MODEL) are much shorter
WHY DO WE STILL LEARN ?
• Industry legacy
– Many commercial entities have deployed SAS since 1970s
– However, note that these deployments are typically ‘click and
run’
• SAS is pretty much focused on pre-building applications to target
industry needs (e.g. revenue management)
• Jobs requiring coding in SAS are technical in nature and may have little
to do with data analysis
– You may get a job in SAS developing statistical applications
– You may get a data scientist job at a mature commercial/public entity
that uses SAS
ARCHITECTURE DIFFERENCES I
• A primer to software-hardware interaction
• What happens when you execute a piece of code?
– CPU computes according to programming logic
• Operators: + - / x
• Logic: if/then/else
• Flow: while x < 1
– Where does it store and read data and steps?
• Random Access Memory (RAM)
• Requires unique physical locations on the hardware (memory addresses)
– Since RAM is like a storage, can we use disk drives in place of RAM?
• Yes, OS X and Windows are doing that (virtual memory/pagefile/swap)
• But they only use it for very specific situations, why?
– RAM is 33x faster than SSD, which is 100x faster than HDD
– Disk drives have limited writes (SSD will wear out in 2 months)
– CPUs can only pre-load data into its cache (small temporary storage area) from RAM
ARCHITECTURE DIFFERENCES II
• An average laptop has 4GB of RAM
– IBES detailed earnings estimates (DETU_EPSUS) is 2.2GB in size
– CRSP daily stock file (DSF) is 3.3GB in size
– Compustat fundamental annual (FUNDA) is 5.5GB in size
• The strategy is to make sure that the size of the required
dataset for RAM-only computation is at the bare minimum
– RAM-only computation: e.g. complex econometrics procedures
• Database software helps us to munge (manipulate) data before
we reach this computation stage
– E.g. MySQL, Microsoft SQL, MongoDB, IBM Oracle
– They perform CRUD (create, read, update and delete) using disk
drive storage space
• Computation is still done by CPU and RAM, but intermediate and
final results are written to disk space
ARCHITECTURE DIFFERENCES III
• SAS is a platform that combines database and programming • Open-source platforms are free, flexible, easily customizable
logic and has a huge support community
– However, users have to set up their own environment
– Users do not need to set up each component separately
– Generally deploy a popular interpreter as ‘glue’ platform
– Most of our working time on SAS involves data munging (e.g. Python, Ruby)
(database-like syntax)
– Separately install database servers
– Running procedures generally involve invoking existing SAS
– Import user-contributed data analytics codes (pandas,
functions (corr, reg, means, summary, univariate etc)
scikit, scipy, numpy) or BYOC (bring your own codes)
RDBS VS NOSQL
• Relation-based Database Systems
– All SQL (Structured Query Language) variants, Oracle, SAS etc
– Once properly setup and indexed
• Very fast and stable
– Good support and long track record
• NoSQL Databases
– Stands for “Not Only SQL”
– MongoDB, Hadoop etc
– Very flexible
– Collections instead of datatype (e.g. store mp3 as one variable)
– Easy to work on insanely huge datasets via clustered servers (distributed computing)
• Remote database server is very useful
– One source, no sync issues
– Uses the computing resources of the remote system for CRUD
– Frees up your local computing resources for other matters
• SAS can use other RDBS as well (via ODBC drivers)
BASIC
PROGRAMMING
LOGIC
PROGRAMMING LOGIC IS UNIVERSAL
• Expressions
– Types (integer, float, string, objects)
– Operators (arithmetic)
• Variables and assignment
• Functions
• Boolean
• Flow control
• Application
T HE M O ST
IM P O R T AN T S TE P
IN D A TA
A NA L Y S IS
BEFORE YOU EVEN START CODING:
VISUALIZE YOUR DATASET SCHEMA!
PROC SQL
WHAT IS SQL?
• SQL stands for Structured Query Language
– It is an ANSI (American National Standards Institute) standard
– SQL is universal across SQL/SQL-supported platforms (syntax may
vary)
• It consists of
– Data definition language
• Creating tables with pre-defined data structures (namely n x m with a
specific datatype for each column)
– Recall the ‘types’ that we have seen during the Python hands-on (integer,
float, string)
• Like SAS, the length and datatype must be pre-defined and all data
entries must adhere to it, until modified
– Integer: int, bigint
– String: length determined by number of characters (SAS max = 65535)
– Dates: date formats
– Float: single or double floating point

– Data manipulation language

• Used for selecting, inserting, deleting and updating data
PROC SQL IN SAS
• PROC SQL is not necessarily better than DATASTEP
– DATASTEP processes data row by row
• Sounds primitive and tedious
• But extremely robust
– DATASTEP cannot be ‘chained’
• May end up creating many intermediate tables
• But extremely robust
– SQL is shorter, more elegant and easier to maintain
• But eats up a lot more resources (e.g. joins require pre-loading)
YOU WILL USE THESE PROC SQL
STATEMENTS ALL THE TIME
• Create
– Make a new dataset
• Select
– Case
– Aggregation functions
• Groupby
– The axis for aggregation functions
• Where
– Slicing data
• Having
– Filters observations
• Order by
– Sorts data
• Joins
– Combine tables
– Left, right, inner, outer, full
– Multiple joins
IM P O R TA NT
P R O CE D U RE S IN
F I NA NCI A L
S TU D IE S
RUNNING FIXED EFFECTS AND ERROR
CORRECTIONS IN SAS
• PROC REG
– Remember to correct for white’s heteroskesdacity by invoking /hcc
proc reg data=car;
• White’s hetero-consistent correction
model car=sue3 bm size dispersion
• Similar to ROBUST in Stata
2002d 2003d 2004d 2005d 2006d
– Invoke /stb to get standardized coefficients to compare effects between
independent variables 2007d 2008d 2009d 2010d 2011d
– Not easy to run fixed effects (levels) 2012d 2013d/hcc stb;
• Requires dummy variables for each level run;
quit;
• PROC SURVEYREG
– Easy to invoke fixed effects
• Use CLASS proc surveyreg data = car;
– White’s hetero-consistent correction is not sufficient cluster date_id;
– Need clustered standard errors class quarter_id;
• Recall: white’s errors relate to residuals across all observations (across groups) model car = sue3 bm size dispersion
• In panel data regressions, we have within groups errors quarter_id /solution adjrsq;
• Invoke CLUSTER to correct for this run;
quit;
RUNNING FIXED EFFECTS AND ERROR
CORRECTIONS IN SAS
• PROC MODEL
– Correcting for time-series errors (NEWEY-WEST)
– Use gmm in proc model

proc model data = drift_portfolio;

endo port_ret_ewrf;
exog mktrf smb hml umd;
instruments _exog_;
parms b0 b1 b2 b3 b4;
port_ret_ewrf = b0 + b1*mktrf + b2*smb + b3*hml
+ b4*umd;
fit port_ret_ewrf /gmm kernel=(bart,5,0) vardef=n;
run;
quit;
INSTRUMENTAL VARIABLE
REGRESSION IN STATA
• Almost remember: Exclusion restriction!
– Instrument should affect X, but not Y when X is held constant
– In other words, instrument should affect X but not the error term in the non-instrumented OLS
• To perform IV regressions in Stata
– Use ivregress depvar indepvar_excl_endogeneous_x (endogeneous_x = instruments)
• Fixed effects and error correction
– Always remember to run your models with error corrections (ROBUST or CLUSTER()
– Fixed effects: append i. in front of the variable ([Link] [Link])
• Many user-created stata modules do not support this
• The workaround is to create dummy variables for each fixed effect
RUNNING TOBIT MODELS IN STATA
• Selection models (heckman; Tobit-2)
• Use heckman <2nd stage model>, select(<1st stage probit model>)
• Uses MLE (numerical method) as default: More efficient
• Invoke twostep option to run it the original way
• Treatment models (Tobit-5)
– Use treatreg <2nd stage model>, treat(<1st stage probit model>)
• Multinomial selection models
– What happens if your 1 st stage model is a logit?
– Use user-created module SELMLOG
• Fixed effects and error correction
– Always remember to run your models with error corrections (ROBUST or CLUSTER()
– Fixed effects: append i. in front of the variable ([Link] [Link])
• Many user-created stata modules do not support this
• The workaround is to create dummy variables for each fixed effect
CALENDAR TIME PORTFOLIO
• First step: Identify the innovative characteristic and form a buy-sell
portfolio
– Create a risk portfolio (similar to SMB, HML)
– In event studies, group stocks by the innovative characteristics that you
have found, e.g.
• Buy firms who announces a new CEO that has beard
• Sell firms who announces one that does not

• Second step: Line up events and calculate daily returns of portfolio

– Line up the events along the calendar time
• Determine the boundaries of your calendar time
• e.g. start at the earliest event date + 0, end at last event date + 90
CALENDAR TIME PORTFOLIO
• Second step: Line up events and calculate daily returns of portfolio
– <continued>
– Determine how long you want to hold for each stock
• e.g. buy at announcement date + 2, sell at announcement date + 90
– For each day in the calendar time:
• Compute the daily returns of all the stocks that are being held in the portfolio
– Remember to adjust your prices with CFACPR
• Use 2 weighting measures: Equally-weighted, and value-weighted
– Remember to adjust your shares outstanding (SHROUT) with CFACSHR

• Third step: Use market returns for days with no stocks

– Replace with market returns for calendar days in which there are no stocks in your
portfolio
CALENDAR TIME PORTFOLIO
• Fourth step: Download and convert the 5 factors (from WRDS FF
database or authors’ websites)
– 5 factors (all in WRDS)
• Risk free rate (RF)
• Market returns (MKT)
• Small minus Big (SMB)
• High minus Low (HML)
• Momentum (UMD)
• Liquidity (Pastor-Stambaugh LIQ)
– For earnings surprise/ drift: Check against Satka’s liquidity measure as well
CALENDAR TIME PORTFOLIO
• Final step: Merge the 5 factor dataset to portfolio returns dataset by date
– Remember to adjust 2 things
• Portfolio return minus RF (LHS)
• Market premium MKT – RF (RHS)
– Use PROC MODEL (for NEWEY-WEST correction) to regress
• (Portfolio returns – RF) = alpha + b1 * (MKT – RF) + b2 * SMB + b3 * HML + b4 * UMD + b5
* LIQ
– Significant alphas implies drift (i.e. market did not fully incorporate information)
• Direction tells us if they are wrong or under-reacted or over-reacted
– Non-significant alphas implies 1. market is correct, or 2. market is not aware of this risk
factor (note: this is hard to sell)
NATURAL
LANGUAGE
PROGRAMMING
MAC H I NE L EARNI NG
LETS HAVE A LITTLE EXERCISE
• This is PM’s national day
message
• I’ll show the full message in
the next slide (only 1,106
words)
• Give an estimate on how
much time will it take you to
– on a scale of 1 to 10, tell
me what is the proportion
of sentiment to neutral
words
– on a scale of 1 to 10, tell
me what is PM’s positivity
PM’S NATIONAL DAY MESSAGE
My Fellow Singaporeans,
50 years ago, on this very night, Singapore was on the eve of a momentous change. The Cabinet had already signed the Separation Agreement. The Government Printers were busy printing
the Separation Agreement and the Proclamation of Independence in a special Government Gazette. The Commissioner of Police and the Commander of the army units had been told by the
Malaysian Government to take orders from the new government the next day. But all this happened in strict secrecy. Our forefathers went to bed oblivious of what was about to happen, still
for the time being citizens of Malaysia. Then morning came. The 9th of August 1965. Our world changed. At 10 a.m., a radio announcer read the Proclamation. Singapore had left Malaysia and
would "forever be a sovereign, democratic and independent nation". The Republic of Singapore was born.
People were apprehensive. No one knew if we could make it on our own. Our economy was not yet viable, much less vibrant. We had practically no resources, and no independent armed
forces. Around noon on that first day, Mr Lee Kuan Yew gave a press conference on TV. He broke down halfway, unable to contain his emotions. It was, he said, "a moment of anguish". But that
moment of anguish turned into a lifetime of determination to forge a path for this island nation. At the end of the press conference, Mr Lee made a promise to Singaporeans. He said: "We are
going to be a multi-racial nation in Singapore. We will set an example. This is not a Malay nation; this is not a Chinese nation; this is not an Indian nation. Everyone will have his place, equal:
language, culture, religion."
From that break, we began building a nation. And what a journey it has been. It started with the first generation of leaders convincing our pioneer generation that Singapore could succeed as a
sovereign country. Together, leaders and the people – the lions and the lion-hearted – fought with unwavering determination to secure our foundations. After them, younger generations picked
up the baton and took Singapore further.
Year after year, Singapore progressed. Along the way we overcame many problems – the British withdrawal in 1971, the Oil Crisis in 1973, SARS, the Asian Financial Crisis, and then the
Global Financial Crisis. We grew our economy and created jobs, built homes, schools, hospitals and parks. We built a nation. Year after year, we have kept the promises that Mr Lee Kuan Yew
made on the 9th of August 1965: that we will be "one united people, regardless of race, language or religion"; that we will always have a bright future ahead of us. Therefore on our 50th
birthday, we have ample reason to celebrate.
Let us celebrate 50 years of peace and security, underwritten by the blood and sweat of generations of NSmen. Let us celebrate how we turned vulnerabilities into strengths. How a
struggling economy with no domestic market made the world our market and created jobs for our people. How without any domestic hinterland, we made PSA and Changi Airport the best in
the world. How from being utterly dependent on Johor for water, we turned the whole island into one catchment area, and developed NEWater. How while we had no natural resources, we
educated every Singaporean and created opportunities for their talents to thrive. We have proven that together, we are greater than the sum of our parts. Most of all letus celebrate how we
journeyed from Third World to First, as one united people, leaving no one behind. Every citizen has benefitted from Singapore's progress. Life has improved for all – for Chinese, Malays, Indians
and Eurasians; for blue collar as well as white collar workers; for HDB as well as condominium dwellers. We are a nation of home owners. Everyone has opportunities to improve themselves.
Everyone can look forward to a brighter future.
At 50 years, as we stand at a high base camp, we look back and marvel how far we have come. We are grateful to those who made it happen. From this base camp, we can also look forward
to new peaks ahead. The journey ahead is uncharted. But we must press on, because we aspire to do better for ourselves and our children. We know that we will get there, because we will
always be there for one another. We are stronger as one people. For example, we instinctively gather to lift a truck to save someone trapped underneath. Even if the music fails, we go on
singing the National Anthem with gusto. We are proud of our past and confident of our future. Together we believe in Singapore; together we belong to Singapore; together, we are Singapore.
I am speaking to you from Victoria Concert Hall, a place that holds special significance in Singapore's history. In 1954, this was called the Victoria Memorial Hall. It was here that Mr Lee Kuan
Yew launched the People's Action Party, and inaugurated the long struggle for a fair and just society. It was here in 1958 that "Majulah Singapura" was first performed. It was at the Padang
nearby, after independence, that we held our National Day Parades, and sang "Majulah Singapura" together as a nation. 50 years on, on our Golden Jubilee, we will gather again at the Padang.
We will sing "Majulah Singapura" proudly, and recite the National Pledge. We will rejoice in the success of our last five decades, and commit ourselves anew to work together as one united
people, regardless of race, language or religion, to build Singapore, so as to achieve happiness, prosperity, and progress for our nation.
Happy 50th National Day!
NATURAL LANGUAGE PROGRAMMING
• For purpose of demonstration, I construct the following naïve analytical equations
– Sentimentality
• Number of sentiment words/ Number of !stop-words
– Positivity
• Positive words minus negative words / Number of sentiment words
– Inclusiveness
• Number of [we, our, ours, us] / Number of [I, my, me, mine]

• Data source
– Harvard-IV word tags
– Semantic directions: (positive, negative, strong, weak, active, passive)
– Words of pleasure, pain, virtue and vice
– Words of overstatement, understatement, presence and absence of emotional expressiveness
– and 12 other categories…
FOLLOW-ON QUESTIONS
• We know that PM writes his own speeches, so its both interesting and important to analyze
further
• How does his speech pattern differ from say… Donald Trump’s campaign announcement?
PM Trump
Less sentiment words
Less positive
Uses more first-person pronouns
– Why the difference?
• Disposition and context

• Has PM’s speech patterns changed?

2013 2014 2015

– More importantly, why? How has the context changed?

FOLLOW-ON QUESTIONS Technical exposition:

• Is PM talking about the same stuff?

– I use cosine similarity to measure how similar the three
documents are
• For demo purpose, I choose a simple feature method, namely the
number of occurrence per word used in each document
– Results show that
• 2013 and 2014 have a similarity index of 51.7
• 2014 and 2015 are very dissimilar, with a index of only 29.3
• 2015 is generally very dissimilar to 2013 and 2014
o X,Y, Z are features
o X,Y, Z = “nation”, “income”,
“family”
o A, B are two speeches (‘13, ‘14)
o Number of feature words
determines the direction and
– Again, what are the differences in the context? length of vectors A, B
o We usually use more complex
features that captures contexts
more tightly
STRUCTURED VS UNSTRUCTURED
DATA
• Structured data
– Structured data has a defined data model
– This is the basis of databases
STRUCTURED VS UNSTRUCTURED
DATA
• Structured data
– Database software allow huge amounts of data to be stored
• Easy to manipulate using CRUD operations (Create, read, update and delete)

SQL
- Structured Query Language
- One of the most popular database

• Databases
– SQL schemas are structured in a 2D table, with strict datatype formatting
• Integers, float, datetime or text
– Data models have increased in complexity with the explosion of data population
– To deal with massive amounts of data and types, NoSQL databases were created
• Not-only SQL
• These databases can store and reference objects (i.e audio, video and unstructured file types)
• Map/reduce algorithms allow massive amounts of data to be delegated, computed and aggregated over clusters of
computers
DEALING WITH UNSTRUCTURED DATA
• What are unstructured data?
– Data models that are not well-defined

Unstructured text:

Peter lives at 123 Holland Road, mobile number 81811188. A banker by profession, he owns a Ferrari.

name address mobile occupation car model

Structured data model

– Other types of common unstructured data: audio, video

– Dealing with unstructured data
• Parse and structure them into data models that supports the intent of your use-cases
• Text: Use regex, machine learning to extract information from text strings
• Audio: Convert into transcripts and parse as text, convert vocal signatures into data
• Video: Use machine learning to detect wanted features in each frame (like a photo)
NATURAL LANGUAGE
• We have seen two methods of analysis on natural language
– Dictionary
– Similarity
• Dictionary method
– Split a sentence into words and map each word to a pre-defined category
• Harvard-IV-4, Lasswell
– This is the easiest, fastest and relatively accurate method
– Improvements to this method has resulted in context specific dictionaries
• Loughran-McDonald
– Training context specificity is possible if there are clear outcomes that we can associate the use of
words with
• Equity prices increase/decrease = presence of [words] and absence of [words] in [news/press releases]
• With a sufficient large number of observations, we will be able to strip out those words that have strong
positive and negative effects to equity prices
• Same thing with fraudulent words, we can use probability of litigation cases and success rates as outcomes
NATURAL LANGUAGE Technical exposition:

• Similarity
– In our demo, we chose to use a naïve method via the
number of word occurrence
– In real life cases, we would first determine the unique
features of the document that we want to compare
others to:
• Idiosyncrasies, grammar usage, vocabulary, structures etc
– Then we calculate the values of these features for the o X,Y, Z are features
source and target documents o X,Y, Z = “nation”, “income”,
– We then use cosine similarity to measure the similarity of “family”
o A, B are two speeches (‘13, ‘14)
each target with source
o Number of feature words
determines the direction and
length of vectors A, B
o We usually use more complex
features that captures contexts
more tightly
OTHER USEFUL NATURAL LANGUAGE
PROGRAMMING TECHNIQUES
• Naïve Bayesian/Maximum Entropy
– Opponents to the dictionary method say that it is naïve
• What about negation, sarcasm, nuances and rare words?
– Naïve bayesian and maximum entropy classifiers are a type of machine learning
• First, we obtain a training set of transcripts (can be blog, forum etc)
• Then for each sentence, we manually read and classify them into target features
– E.g. if we want to detect spam, then we read and classify each sentence if its spam or not
– This can be done for any feature we want (gender, age, positivity, sarcasm etc)
• The algorithm then computes the unconditional (naïve bayesian) or joint (max entropy) probabilities of words that
are associated with each feature
– E.g. [‘lottery’, ‘Strike’, ‘rich’, ‘fast’] predicts spam 100%
– Features need not be the words in the sentence, but things such as
• Named entities (names, places, companies, countries, geography etc)
• Parts of speech (Pronouns, verbs, nouns, preposition etc)
• Logic (‘happy’ occurs within 3 words from the beginning:True/False)
OTHER USEFUL NATURAL LANGUAGE
PROGRAMMING TECHNIQUES
• Topic identification
– Given a large amount of articles, how do we know what the major topics are?
– Method #1: Latent Semantic Analysis/ Latent Dirichlet Allocation
• Compute the word counts for each article (vector) and combine them into a matrix
• Use Singular Value Decomposition to preserve the similarity structure while reducing dimensionality
• Use cosine similarity to compute the similarity between articles
• Results (using wikipedia corpus) look like this

– Topic #0 appears to be about geography/landscape

– Topic #1 is about sports
– Topic #2 is about judiciary
• Using clustering techniques, we will be able to know which article falls into which topics
LSA VISUAL SAMPLE
(F ro m ht tp:// w w w .pu ff in war ellc .com /in dex .p hp /n ew s-an d-ar ticles /art icles/ 33 -late nt -sem an tic -an alysis -
tu tor ial.h tm l?sta rt= 1)

• Given the following articles:

• Apply LSA to get the following

OTHER USEFUL NATURAL LANGUAGE
PROGRAMMING TECHNIQUES
• Topic identification Topic Words (with td-idf weights)
– Method #2: Explicit Semantic Analysis missile technology = 0.34 army + 0.21 flight + ...
• This method is especially useful in our work
• Select a set of articles that have titles/headings
– Wikipedia pages have titles followed by body army = 0.34 missile technology + ...
– News releases have heading followed by body flight = 0.21 missile technology + ...
– Forums have topic followed by body

• Using the NLP technique TD-IDF (term frequency-inverse document frequency)

– TD-IDF computes the word (term) frequency for each article, scaled by the popularity of that word in the entire
sample
– Since each article has a human readable label (topic), we can associate the TD-IDF matrix to each label
– Inversing the topics and words for the entire sample, we get a vector (list) of topics per word
• Given any combination of words, we just need to sum the topic vectors of the words to obtain the
probabilities of the topics that this combination of word is associated with
GRAPHICS
MAC H I NE L EARNI NG
MACHINE LEARNING AND GRAPHICS
• Video is simply a scrolling of multiple frames in quick succession
– Standard photo analytical procedures can be performed on each frame
– Requires much more computing power compared to analyzing still photos
• One basic machine learning technique to detect similar faces
– Images are made up of many pixels, each of which is encoded with a color code
• 1024 x 768 means length and breadth of 1024 and 768 pixels, with a total of 786, 432 pixels
– Collect many mugshots
• Scale them to the same resolution
• Position the faces such that their features are positioned in approximately the same areas of the shot
– For each mugshot
• Read the pixel values row by row and represent them in one line (vector)
• Combine all vectors into a matrix
MACHINE LEARNING AND GRAPHICS
• Use Principal Component Analysis, a common
dimension-reducing technique to extract the
features (now pixel value) that represents the
most variations
– The resulting data is called eigenface
• Training the sample
– Method #1:
• Calculate the mean of the eigenface matrix (this
is the benchmark face)
• Calculate the eigenvector and eigenvalues for
each eigenface to the mean (this is like the
fingerprint for each eigenface)
– Method #2 (recommended):
• Use Support Vector Machine (SVM) to calculate
the maximum vector values that separate one
eigenface from another
Image from [Link]
MACHINE LEARNING AND GRAPHICS
• With the trained sample, you can feed any
mugshot to the program and it will
– Convert the mugshot to eigenface
– Compare the eigenface to the trained sample
– Clustering methods to determine which of the
trained sample is the best match
• By now, you should have realized that machine
learning is generally about
– Extracting features from a sample dataset
– Training the machine to effectively differentiate
one feature from another
– Once trained, you can supply the machine with a
new observation and it will try to match it with
one of the trained sample, using features that it
has learnt
Image from [Link]
SOCIAL
NET WORKS
SOCIAL NETWORK CENTRALITY
(GRAPH THEORY)
• We represent social networks as graphs
– Board of directors
– Firms and other entities
– Friends
• Nodes represent an entity (can be a
business, organization or individual)
• Lines represent a connection between 2
nodes
– Worked at the same firm, studied in the
same school etc
SOCIAL NETWORK CENTRALITY
(GRAPH THEORY)
• With graphs, we can compute some
useful metrics in our work
– Cliques/ Clusters
– Connectivity
• Largest connected component
• Proportion of graph with >k neighbours
– Influence/Importance
• Eigenvector: Connections to highly
connected nodes are more valuable
• Closeness: Ability to reach entire
network (through connections) in the
shortest amount of time
• Betweenness: One of a few that connects
(bridges) two clusters
SOCIAL NETWORK CENTRALITY
(GRAPH THEORY)
• With graphs, we can compute a variety of
useful metrics
– Influence/Importance
• Authority: Generates information
• Hubs: Relays information
BOARD NETWORK

• HK and SG Board Network: Cliques

QUESTIONS AND
FURTHER
DISCUSSIONS