Skip to content

Time series prediction and text analysis using Keras LSTM, plus clustering, association rules mining

License

Notifications You must be signed in to change notification settings

moment-of-peace/EventForecast

Repository files navigation

-----------------------

website: https://github.com/moment-of-peace/EventForecast

EventForecast

A group data mining project using deep learning (LSTM) aiming at predicting the probability of occurence of particular events and the popularity of events. This project is granted "The Best Data Mining Project Award" of the University of Queensland in 2017


dataset

Source data: https://www.gdeltproject.org/data.html

Crawled news data on google drive: https://drive.google.com/file/d/0B--MjMVnQr09SmJ2VGh5VkstR2c/view?usp=sharing

Processed data: https://drive.google.com/drive/folders/0B_Qs_6HNIHS9Qk1hMzZ1c3VWOE0?usp=sharing


Occurrence Prediction

Environment: numpy, scipy, tensorflow, keras, h5py, matplotlib

Require at leat one "attr-county" folder provided by processed data link above (need to unzip first)

Use following command to run (-p is compulsory, others are optional):

python3 rnn_model.py [option parameter]

-p: the path of precessed events records for a certain country (any "attr-country" folder in the processed data link)

-s: step size, how long history to consider

-a: how many days look ahead (which day in the future to predict)

-e: training epochs

Popularity (hot events) Prediction

Environment: numpy, scipy, tensorflow, keras, nltk, h5py, gensim.

Require folder "news_50_num" (need to unzip first), and word embedding files "vocab_event100.pkl", "weights_event100.npy" provided by the processed data link above

Use following command to run (all options are optional):

python3 rnn_text.py [option parameter]

-b: batch

-l: cutting length, i.e. how many words in the news to consider

-e: training epochs


Cluster

Environment: R language:

1.copy the exact data in data/test.xlsx (X to AP)

2.z=read.table("clipboard",header=T)//import data in to z

3.km <- kmeans(z[,1:20], 3)//use build-in Kmeans function

4.km//print km results


Other files

preprocessing.py: general preprocessing, including removing useless data, extract and count events and so on.

nlp_preprocessing.py: special preprocessing for text analysis, including removing common words, stemming, and so on

hot_news_php/ : files for the webpage application about hot news recommendation

association_rule/ : codes association rules mining

cluster/ : files and results for clustering

About

Time series prediction and text analysis using Keras LSTM, plus clustering, association rules mining

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 4

  •  
  •  
  •  
  •