Coda Twitter Discovery Trends

Have you ever gotten frustrated finding the content you want in twitter?

Comment

Inspiration

I have been frustrated trying to find helpful educational content around specific topics in twitter and can't seem to find the right people to follow. Thought it would be great to have a place to analyze popular tweets for a topic like "machine learning" and create my own dashboard around trends. This is what inspired me to create the Coda Twitter Discovery Trend pack.

What it does

So you can create a Trend sync table, where you input information like the trend you would like to follow e.g "Machine learning", your email address, the date you want to look back etc. Once the table finishes syncing you will start to see the tweets related to this trend, and the sync table would grow from when you started streaming tweets. Create your own trend dashboard in coda based off this sync table.

You will get notified by email once a day with the top 5 trending tweets for the topic you specified.

An added sync table called Recent Search will search tweets for a topic in the past 7 days.

How we built it

Created two sync tables with the coda pack sdk. The Trend sync table would query a Tweet Rule Service to create rules for a Twitter Stream, along with querying a Tweet Trend Search service.

For the services I created in the backend, I firstly forked off the gcloud-toolkit-filtered-stream to start with which gave some good examples on how to stream tweets from the Twitter stream api service. From there I created my own services to be queried by the coda pack extension.

The Trend sync table would create a Trend to stream via the Tweet Rule service which would create a rule and tag id via the Twitter Filtered Stream Rules API. The Streamer service would pull tweets from the Twitter Filtered Stream API and push them to a Google Pub/Sub. A google cloud scheduler would then invoke the Loader service which would pull tweets from the Google Pub/Sub and load the tweets on the BigQuery dataset. The Coda pack would also query the Tweet Trend search service which would load tweets via BigQuery.

Another google cloud scheduler would invoke the Tweet Mail Service once a day to send out emails to the people who have created trends via the coda pack. This service would check Google Firebase for the tag ids and corresponding email address, load all the relevant tweets for the tag id and send the top 5 to each person.

The coda pack had another sync table which queried the Twitter recent search api via the Recent Search service deployed on google app engine. (this would give recent tweets in past 7 days)

Have a look at the architecture in the media for this project.

Challenges we ran into

While experimenting, keeping track of the relevant schemas in place from the coda sync table schema to the bigquery schema was challenging sometimes. However the coda sync table schema was quite flexible.

Being able to link the email address given by the user and the trends they have created to stream via the sync table was an interesting challenge. Decided to use a tag id which can be created with the Twitter Stream rule api, this proved to be quite valuable since I could link this in the Google BigQuery schema along with linking it to the email address of the user.

Accomplishments that we're proud of

Having a trend sync table I can now create for different topics I want to follow and analyse, also the google job scheduler that sends an email once a days for the top trending tweets of that topic is quite useful.

I've already found out some interesting content and twitter users from creating this coda pack so am pretty happy with it.

Learning more about deploying systems on google cloud by using google job scheduler, pub sub, BigQuery and Firebase are all useful tools to have.

What we learned

Make sure to think about the end to end user experience early on and how you will architect your system to achieve that. Changing schema to suit a preferred user experience can be a bit of a pain. Also know what data all your services need and how coda pack syncs and gathers it data is very important.

What's next for Twitter Trends

Improve the mail service so it can run email batches more efficiently, add action buttons in the sync table to bookmark tweets. Add more columns in the sync table so people can better analyze trends they are following. Improve the email template being sent to users (I am not the best front end engineer).

Built With

Share this project:

Updates