S𝘤à𝑙ẫ streaming programming final pr💡ject

by 🅐🅛🅔🅧🅐🅝🅓🅔🅡 🅞🅝🅑🅨🅢🅗

This is a final project of Streaming programming with Scala at UCU. The aim of this project is to develop services for streaming data to Kafka topics then combine them using Kafka stream join.

𝑰dea 💭

The idea of the project is to get real stream of tweets and stock prices for given stock symbols, join them by symbol and forward to another Kafka topic for future processing.

Diagram shows rough architecture:

ℙroject 𝕊tructure

twitter-service - services gets real time stream of tweets with given keywords, all messages go to kafka tweets topic.
stocks-service - services gets real time stocks prices from IEX by given stock symbols, all prices go to kafka stocks topic.
combiner-service - service pulls messages from tweets and stocks, join them by stock symbol and sends to enriched topic.

𝕊etup 𝚕ocal 🅺afka 𝔠luster

This command will start zookeeper and kafka clusters. Default port is 9092.

make kafka-start

Shut down cluster:

make kafka-stop

𝕊etup 𝓼ervices 🛰

1. Create stocks-credentials.env

TOKEN=sk_0000000000000000000000000

Token should be requested from IEX

2. Create twitter-credentials.env

ACCESS_TOKEN_SECRET=
ACCESS_TOKEN=
CONSUMER_SECRET=
CONSUMER_KEY=

Credentials should be requested from Twitter Developer

3. Create topics in Kafka

make kafka-create-topics

4. Run services

make services-start

# to stop
make services-stop

𝐕iew 𝓂essages ⅰn 🅺afka ᵗopics 📬

1. Install kaf tool github

make install-kaf

3. View topics

kaf topics

2. View topic's messages

kaf consume tweets
kaf consume stocks
kaf consume combined

Ｂuild 𝒾mages 📦

All images are build automatically from master branch by DockerHub:

• alexanderonbysh/twitter-service:latest
• alexanderonbysh/stocks-service:latest
• alexanderonbysh/combiner-service:latest

If you want to update code just rebuild docker image:

cd twitter-service
docker build -t twitter-service .

𝔑otes 📝

Ẅhat 𝗖ould 𝘣e 🄘mproved 🚀

• move data object like Tweet, Price, EnrichedTweet to common package 🙊
• handle rate limits for Tweeter and IEX API endpoints 🙉
• Tests 🙈