Methodology
The project follows a systematic, data-driven approach to evaluate how sentiment in
financial news influences stock market trends. The methodology is divided into the following
key stages:
1. Data Collection
Sources: Financial news articles, headlines, press releases, and reports from trusted
platforms (e.g., Bloomberg, Reuters, Yahoo Finance).
Stock Data: Historical stock prices, volume, and indicators from APIs like Yahoo Finance,
Alpha Vantage, or NSE/BSE datasets.
Timeframe: A selected window (e.g., 1 year or 3 years) to analyze market response over
time.
2. Data Preprocessing
Text Cleaning: Removal of stop words, special characters, numbers, and redundant
whitespace.
Tokenization: Breaking text into individual words/tokens.
Lemmatization/Stemming: Reducing words to their root forms to ensure consistency in
sentiment analysis.
3. Sentiment Annotation
Lexicon-Based Approach: Use of financial sentiment dictionaries like Loughran-McDonald to
tag positive, negative, and neutral words.
Machine Learning Classification: Supervised learning models (Naïve Bayes, Logistic
Regression) trained on pre-labeled financial sentiment datasets.
Deep Learning (optional): BERT or LSTM models for contextual sentiment classification using
sentence embeddings.
4. Sentiment Scoring
Assign sentiment polarity scores (e.g., -1 to +1) to each news item.
Calculate Daily Average Sentiment Index to represent aggregated news sentiment per
trading day.
5. Correlation Analysis
Link sentiment scores with corresponding stock performance (daily return, volatility,
volume).
Perform statistical correlation analysis (Pearson/Spearman) to measure the strength and
direction of relationship.
6. Predictive Modelling
Use time-series forecasting models (e.g., ARIMA + sentiment input, LSTM) to predict short-
term stock movement.
Compare models with and without sentiment inputs to evaluate predictive power.
7. Visualization and Interpretation
Plot sentiment trends vs. stock price trends using graphs.
Visualize spikes or dips in sentiment and how they precede or coincide with stock
fluctuations.
Analyze patterns, anomalies, and cause-effect insights.
8. Evaluation Metrics
For Sentiment Classification: Accuracy, Precision, Recall, F1-score.
For Stock Prediction Models: RMSE, MAPE, R² score.
9. Conclusion and Implications
Summarize findings on the effectiveness of sentiment analysis in predicting stock behavior.
Discuss limitations (e.g., noise in news, market irrationality) and potential areas for further
research.