CrowdPulse

Can Twitter predict the stock market? CrowdPulse answers that question with a full machine-learning pipeline and a Bloomberg-inspired live dashboard.

---

What is CrowdPulse?

CrowdPulse is a data science project that tests whether Twitter/X sentiment predicts next-day stock price direction across 20 major tickers. It covers the full pipeline — from raw tweet data and OHLCV prices all the way to a trained classifier and an interactive trading dashboard.

The core question: does crowd emotion move markets, or does it just follow them?

---

Tickers Covered

Sector	Tickers
Big Tech	AAPL, MSFT, GOOGL, META, AMZN
Semiconductors	NVDA, AMD, INTC
EV / Growth	TSLA, SHOP, SNAP, UBER
Streaming / Media	NFLX, DIS
Finance	JPM, BAC, GS, PYPL
Market Index	SPY
Global	BABA

---

Project Structure

CrowdPulse/
├── pipeline.py                  # End-to-end ML pipeline (Steps 1-5)
├── app.py                       # Bloomberg-inspired Dash dashboard
├── fetch_data.py                # Data fetching utilities
│
├── daily_sentiment_returns.csv  # Aggregated daily tweet sentiment per ticker
├── stock_prices.csv             # Daily OHLCV price data (20 tickers)
│
└── outputs/
    ├── merged_data.csv          # Merged sentiment + price dataset (24,700 rows)
    ├── features.csv             # Engineered feature matrix (32 features)
    ├── model.pkl                # Trained Logistic Regression model
    ├── feature_columns.pkl      # Saved feature column names
    ├── metrics.txt              # Full model evaluation report
    └── plots/
        ├── sentiment_over_time.png
        ├── stock_close_over_time.png
        └── sentiment_vs_return.png

---

Pipeline — Step by Step

Step 1 — Merge & Clean

• Joins daily tweet sentiment data with OHLCV stock prices on [ticker, date]
• Drops nulls, filters to the 20 target tickers
• Creates binary target: price_up = 1 if daily_return > 0
• Output: outputs/merged_data.csv — 24,700 rows

Step 2 — Exploratory Data Analysis

Generates three diagnostic plots saved to outputs/plots/:

Plot	What it shows
sentiment_over_time.png	Mean daily sentiment for all 20 tickers over the full date range
stock_close_over_time.png	Normalised close prices (base = 100) to compare relative performance
sentiment_vs_return.png	Scatter of sentiment vs daily return, coloured by price direction

Step 3 — Feature Engineering

32 features are constructed from raw sentiment and price data:

Feature	Description
mean_sentiment	Average VADER compound score for the day
pct_bullish / pct_bearish	Share of positive / negative tweets
net_bullish	pct_bullish - pct_bearish — net crowd leaning
sentiment_momentum	Day-over-day change in sentiment
roll3_sentiment	3-day rolling mean sentiment
roll7_sentiment	7-day rolling mean sentiment
lag1_mean_sentiment	Yesterday's sentiment score
lag1_mean_vader	Yesterday's VADER score
log_tweet_count	Log-transformed tweet volume
ticker_*	One-hot encoded ticker identity (20 columns)

Also runs an Augmented Dickey-Fuller (ADF) stationarity test on each ticker's close price series.

Step 4 — Modelling

• 80/20 chronological train/test split (no data leakage)
• Trains two classifiers with class_weight='balanced':
  • Logistic Regression (max_iter=1000)
  • Random Forest (n_estimators=200)
• Saves winning model to outputs/model.pkl

Step 5 — Evaluation

Full metrics report written to outputs/metrics.txt.

---

Model Results

Overall Performance

Model	Accuracy	Precision	Recall	F1
Logistic Regression	70.7%	66.9%	70.6%	0.687
Random Forest	68.8%	64.1%	71.6%	0.677

Winner: Logistic Regression — F1 = 0.687 vs random baseline of 0.50

Per-Ticker Accuracy (Logistic Regression)

Ticker	Accuracy	Ticker	Accuracy
AMD	74.9%	PYPL	70.9%
META	74.1%	BABA	70.5%
INTC	73.7%	SNAP	70.5%
JPM	73.7%	AAPL	70.0%
NFLX	73.3%	GOOGL	69.2%
SHOP	73.3%	GS	69.2%
NVDA	72.9%	MSFT	69.2%
TSLA	72.1%	BAC	67.2%
AMZN	70.9%	DIS	66.4%
—	—	UBER	66.0%

Top 5 Features by Importance

Rank	Feature	Importance
1	pct_bearish	1.585
2	mean_sentiment	1.487
3	net_bullish	1.487
4	sentiment_momentum	0.875
5	ticker_TSLA	0.676

Conclusion

Twitter sentiment shows moderate predictive power for next-day stock direction. The best model achieves an F1 of 0.69, well above the 0.50 random baseline. Sentiment momentum and rolling averages are the most informative features — short-term crowd sentiment shifts carry real signal. However, F1 remains below 0.80, indicating that sentiment alone is insufficient for reliable trading decisions.

---

Dashboard

CrowdPulse ships with a Bloomberg Terminal-inspired interactive dashboard built with Plotly Dash.

Features

• Live ticker tape — real-time price scrolling across the top
• Interactive candlestick chart — OHLCV data via yfinance with configurable date range
• Sentiment mood chart — rolling average sentiment trend overlaid on price
• Sentiment vs return scatter — visualise the correlation live
• Correlation heatmap — sentiment x price feature correlations across all tickers
• AI Prediction panel — run the trained model on custom sentiment inputs:
  • Select any of the 20 tickers
  • Set sentiment score (-1.0 to +1.0) and % bullish via sliders
  • Get a prediction (UP / DOWN), probability gauge, and plain-English explanation
• Learn section — 5 collapsible explainer panels for non-technical users

Running the Dashboard

# Install dependencies
pip install dash dash-bootstrap-components plotly pandas numpy scikit-learn joblib yfinance statsmodels matplotlib

# Run the pipeline first (generates model + outputs)
python pipeline.py

# Launch the dashboard
python app.py

Then open http://127.0.0.1:8053 in your browser.

---

Setup

Requirements

dash
dash-bootstrap-components
plotly
pandas
numpy
scikit-learn
joblib
yfinance
statsmodels
matplotlib

Install all at once:

pip install dash dash-bootstrap-components plotly pandas numpy scikit-learn joblib yfinance statsmodels matplotlib

Run Order

# 1. Run the pipeline first (required before dashboard)
python pipeline.py

# 2. Launch the dashboard
python app.py

Note: tweets_sentiment.csv (78 MB) is not included in this repo due to GitHub file size limits. The pre-built outputs — model, features, and plots — are included so the dashboard works out of the box without re-running the pipeline.

---

Data Sources

Dataset	Description
tweets_sentiment.csv	Raw daily tweet sentiment aggregates per ticker (VADER-scored)
daily_sentiment_returns.csv	Processed daily sentiment + return signals
stock_prices.csv	Daily OHLCV price data for 20 tickers

Sentiment scores are computed using VADER (Valence Aware Dictionary and sEntiment Reasoner), an NLP tool tuned for social media text. Each tweet receives a compound score from -1 (most negative) to +1 (most positive), aggregated daily per ticker.

---

Tech Stack

Layer	Tools
Data processing	pandas, numpy
NLP / Sentiment	VADER (pre-scored)
Statistical tests	statsmodels (ADF stationarity test)
Machine learning	scikit-learn (Logistic Regression, Random Forest)
Visualisation	matplotlib, plotly
Dashboard	Dash, dash-bootstrap-components
Live market data	yfinance
Model persistence	joblib

---

License

MIT — see LICENSE

---

Built with data, doubt, and too many tweets.