Simon de la Rouviere

Twitter’s Archive and (my) TwimeMachine.

I started TwimeMachine back in late 2009, because it was a horrible user experience to find and read your old tweets (hint: click next, click next, click next).

Over the years it’s evolved, getting help from people like Misha Kvakin (TwimeMachine’s design for a long time), changing codebases (PHP to Django) and finally having to work around Twitter’s recent deprecation of some timeline features.

Even before Twitter’s recent ‘pivot’ away from its API ecosystem, I knew it was something that Twitter eventually just had to implement. It was a nice, fun side project. I could get what I want, other people could use it, and I can learn new skills (Heroku, Django, better javascript, etc). I never banked or spent too much time on it for this reason. I’m okay with that. I think it would be naive to expect something if you fill a gap in a third-party way. Even if the company has a history of buying up great third-party apps (which Twitter of course doesn’t have), it would still be naive to expect anything.

As you may know, Twitter announced the ability to download and view all your old tweets. I haven’t been able to test it as it hasn’t rolled out to me yet. By the looks of it, you request your archive, it mails it to you, and then through a separate HTML file you can read and search your old tweets. It’s got an interesting interface. By downloading the tweets beforehand, you can quickly navigate through the months on the right. This is similar to how TwimeMachine initially functioned before they changed how the timeline API works. You could quickly skip to parts.

So I guess it comes now to an important question: Do I think it will kill TwimeMachine?

No. The user experience is a bit different. I think TwimeMachine would still be better in some use cases. Firstly, requesting your archive and waiting each time for an e-mail, then opening that HTML file is more effort than simply logging in and waiting a few seconds for 3200 tweets to load.

Secondly, the inline interface is (debatably) better for quickly scanning and browsing your past tweets.

Thirdly, and probably the biggest difference: You can read other people’s old tweets.

—-

So does the fact that Twitter have all your tweets outweigh the other benefits? Not entirely, I’d say, but I do think I’ll lose some traffic over it. Initially I suspect traffic might increase due to chatter, but long-term, the people that use it to read their whole archive (or first tweet) will go the Twitter route. Another factor that is detrimental is that more and more people are tweeting, cruising past 3200 tweets.

It will be interesting to see what happens. I’m proud so far that this little side-project has already been used by about 250 000 Twitter users including Paulo Coelho, Cesc Fabregas, Adidas, CNBC, Sega and ESPN amongst others. I have gotten way more out of it than I could’ve dreamed of.

The ad revenue is after all sponsoring my dev account over at app.net. ;)

Changing information consumption habits.

As a person that is very involved with all that the wonderful web provides, it is sometimes a battle to not just dive into endless articles of fascinating news and ideas. The longer the web is around, the more interesting sites and reads I find. It starts to pile up!

It’s easy to get lost in reading reddit, twitter, facebook, G+, hacker news, reader, tumblr etc. Your brain goes off on this tangent of interestingness (fueled by dopamine hits of information). Before you know it, 2 hours have gone past and you haven’t done what you wanted to.

Adam Brault quit Twitter, and in that blog post, he said something that hit home:

I’ve realized how Twitter has made me break up my thoughts into tiny, incomplete, pieces—lots of hanging ideas, lots of incomplete relationships, punctuated by all manner of hanging threads and half-forked paths.

It’s time to change my consumption habits. This is how my mind is starting to think (and want).

There’s a reason why I decided to do my masters on information overload. I wanted to solve this for myself. How can we manage information in this unprecedented 21st century explosion of ideas?

I’ve already started to force myself to take breaks and just to go sit outside. I especially enjoy it in the summer as the sun sets, drinking a whiskey. I have to let go of wanting that constant drip of information.

The biggest change however is one that I’m embarking on right now. I recently got a Galaxy S3 (such a relief from my old HTC Wildfire). I’ve loaded up Pocket and going to use that as my main way to consume new articles. If I’m Hacker News or Twitter and find an interesting link, I’m going to add it to Pocket and leave it for later.

I want to unbundle information consumption to a set device, with set times.

In other words. My macbook needs to function mainly as a production or creative device. When I’m on it, I want my mind to be in a “write thesis”, “code”, “make music”, “e-mail” mode.

When I’m on my phone, it’s for Facebook, Twitter, G+, Tumblr, HN, etc.

To ‘stay’ recent, at first I’m going to try and restrict this to lunch and after ‘work’. At first, I’m only going to do this with feeds and articles. If that goes well, I can perhaps switch off Facebook and Twitter (other two culprits) during my times I want to product. Baby steps.

I’ve also started using Prismatic. It’s gotten great praise for handling news. So far, it’s been amazing. Finding awesome content through it. The interface still needs tweaks, and a native Android app. atm, I save links to Pocket through the browser. They haven’t integrated their “read later” with Pocket yet. I’ve tried to find a work-around, but haven’t found it yet. So, now I’m using Google Reader mainly for blogs that I know I want to read every post of (such as Elezea), which might not surface in Prismatic. I’ve unsubscribed from larger feeds such as TechCrunch.

My ideal flow would thus be to use Prismatic to surface content, save to Pocket and then read it mainly from my phone. If I find great links to share, I can simply add it from Pocket to Buffer and let fly back again into the www!

How has your information consumption habits changed? What is your main setup for reading new news and interesting articles?

Update on some stats about TwimeMachine. Passed 200k user mark.

TwimeMachine is still churning along.

As you might know. I started it in 2009 (then called oldtweets). It was a hack-job I once put online. Surprisingly, it kept getting traction. So I recoded it in PHP and moved it twimemachine.com (early 2010). Then I recoded it in Django and put it up on Heroku at the end of 2011. Then a few months ago, Twitter’s API changed and did a completely new version (also in Django), which is the version you see now.

Without any effort, people keep finding it and using to read/find their old tweets. I’ve added an opt-out checkbox to follow @twimemachine when you sign up at the beginning of the year. It now almost has 30 000 followers. I don’t store anything, so I don’t really know how many people have used. My only metric is Google’s unique visitors. There aren’t any other pages besides logging in and using it, so I use “unique visitors” * “bounce rate” as metric. 63% of people go through (average bounce rate of 37%), with 321k unique visits across the 2 years of growth. So my estimation is that about 202 000 people have used it!

When people follow TwimeMachine, I get separate notifications if people with large followings use it. I’ve had the privilege that accounts such as Paulo Coelho, ESPN, Parents Magazine, SEGA, CNBC, Adidas Originals, LA Times, Danny Zuker, Venturebeat, amongst others have used it.

It’s hosted on Heroku, for which I’m paying nothing (functions in the free zone). The only thing I pay for is for the image files on S3 (amounts to about $0.11 a month). The adsense on the other hand usually makes between $50 - $100 a month. Nice pocket money for future endeavours.

——-

I’ve been thinking what to do with TwimeMachine. It’s intention was never to be big. I’m not trying to put too much effort into if Twitter is just going to do something similar. And I’m especially wary lately to rely too much when building something on another person’s platform. Most notably are all the changes to Twitter’s API, which doesn’t sit too well with me.

On the one side, I’ve been wanting to sell it, get some return on it. But on the other hand, it is still growing without any effort on my part. I can still do great things with an account of 30 000 followers. Been thinking of doing something charity/donation related, but want to find something that is relevant (not just something like charity:water).

So yeah. It’s quite rewarding where it’s been and where it is going. Never expected it to be this useful to so many people!

Thoughts on app.net

I’ve been watching app.net for a while since Dalton’s ‘audacious’ post about. I really liked what he proposed. I was one of the early adopters of Twitter’s API. I started Tweekly.fm in early 2009, being under the first 500 apps. Now there are more than 3 million registered apps.

I loved working with it, and like most other developers were really excited about what it could become. When Twitter announced annotations, I was really excited. This meant anyone could embed any metadata into posts! An example of something I thought I could easily do on top of Twitter’s API was related to Tweekly.fm. I could embed music information into the posts. Anyone could then mine it for their use, contributing back to an ecosystem.

Alas, that never came to fruition. 2010 was the first and last time Twitter held a developer’s conference. I understand their choice, but I don’t like it. Nothing against them. I still use Twitter daily and probably will for quite a while still.

Now that app.net came along and promised what I wanted, I witheld at first, checking it out. I even started contributing to the API spec before I backed. My biggest worry was kickstarting the network effect. I’m not paying for an awesome API if no one is going to use what I build. Dalton and co pulled out the stops and even provided an alpha version on top of the API. Respect. To give them the benefit of the doubt and really wanting something like this, I decided to back. Even if a small community of tech (hipsters) use the API, it is still the worth a year’s tier.

—-

Now that it got funded, I’m really looking forward to the following. I don’t think people realise what amazing things can be built on it, and what is to be expected of it. My favourite API features: Annotations and Filters.

As mentioned previously, annotations allow any post to be embedded with meta information. Now: Filters allow any stream to filtered based on a lot of criteria. Filters are unique to a user and can be taken with them to any app.net app.

So, let me give you a few example of the power of this:

1) Don’t like it when people cross-post to app.net from Twitter or Facebook? Create a filter that discards all tweets with sources from other social networks. Now you have a pure experience.

Orian Marx describes the power of app.net eloquently (via gigaom):

“App.net will combine the simplicity of cloud infrastructure with the power of web frameworks to deliver the best platform for developing social web applications.”

Now take that idea to something as follows:

2) Let’s start a photo sharing app. Instead of having to build my own social infrastructure, I tack onto app.net. When posting photos, I embed captions, urls, etc. Now in the app itself, doesn’t matter who you follow, I can add a filter to ONLY retrieve this apps photos from your existing app.net graph.

Now, going even further, I can add more filters to that stream as well, only retrieving users you actually chose to ‘follow’ on this service if you don’t want to follow your existing app.net graph.

The potentials are endless. Currently developers are only building the normal suite of apps: ios, desktop, etc. Once the API comes out of alpha, we’ll start seeing the so called ‘killer’ apps that use these new API features.

—-

On the other side. I’m not naive to disregard that app.net might still fail. Inside, its an echo chamber: an early adopter community of great people. Convincing people to join this network for $50 a year (if it stays this) is going to be app.net’s biggest problem. I suspect if a killer app comes along, users will pay. However from a UX perspective it might be a bit jarring, wanting to sign up for a service, but then having to resort to paying for ANOTHER service they know nothing about. Perhaps, developers can handle the payments on behalf of app.net for continuity’s sake.

Either way. I am excited about it. I want it to succeed.

I have a Python API wrapper for it out on github that uses the current available alpha API endpoints.

Unless social networks change, they’ll die.

Dalton Caldwell recently posted about his experience using Orkut back in the day. It was a great place, until the community changed away from what he wanted. In the rest of the post, he talks about Critical Mass and Network Effects.

In short: In order for a social site to become sustainable, it needs to have critical mass. Users come back, because enough is going on for them to come back. However, this continues to create value, leading to characteristically hockey-stick growth. This value can’t be sustained due to restraints in how the medium is structured and limits of our own to process information. This is the fun part, the challenge that a lot of these sites are struggling with (Facebook, Twitter, etc). Users won’t stop becoming connected to more users, whether it is close new friends, or no one you know (ie a new follower on Twitter). In order to sustain critical mass discourse, some things need to change.

The one’s that been stuck in the back of my mind, is to decrease the amount of “entropy” created by the increase of information in the network.

The larger your graph becomes and the more information you share, mean you start looking at sharing differently. ie, with each new piece of information being added to the network (in form of updates, photos or new connections), it adds value, but returns start to diminish. More information means more to consider.

Let me try and explain by a metaphor (and perhaps a lame platitude):

To get a car driving at top speed, it is easier and faster to start from 1st gear. Once it kicks in, acceleration increases, changing gears (value is still being added), top speed increases… up to a point where any addition to the top speed means new problems arise (that wasn’t evident before). Driving faster means greater possibility of accident and more considerations to take into account. Instead, the car and driver tend toward an equilibrium of speed that is sustainable.

The same with social networks, however these sites don’t look at how to trend towards critical mass discourse equilibrium. Recently however, it’s begun to change. There is less informational baggage hanging about.

Facebook, with it’s new timeline feature, summarises old content. Unless you know where to find it, it’s nigh impossible to just scroll through it all. Users who you haven’t been involved don’t show up in your feed, everything is becoming more and more filtered. Facebook is running at critical mass. It accelerated from a small college site, adding features along the way and growing into the behemoth it is today.

Twitter is a great example as well. The system is thankfully still relatively simple (not too much to consider). It is and always has been focused on “What’s happening”. This is fleeting, and it shows. There is less information bogging new interactions down. It is much easier to run at critical mass with a system like Twitter, however they aren’t doing any filtering. The result shows: critical mass discourse occurs in groups less than 150 (http://arxiv.org/pdf/1105.5170).

What it comes down to is that running a complex system at critical mass is difficult. You won’t know whether the system you built will be the cause of its own death, because just as the car shows its new troubles at higher speeds. Trying to change such a system at such a stage can cause user confusion, so instead they’ll just have to change constantly.

P.S. I’m currently researching information overload on microblogging services. Sometimes it is just nice to write in a non-academic way and be able to use unbacked opinions (that hopefully will become scientifically backed opinions).

Twitter’s summarizing my follower notifications

So. This might have been happening already, but I recently discovered a change in the e-mails Twitter send me when one of my accounts gets followers. I assume it is for accounts that receive frequent followers?

@twimemachine has been receiving between 100-200 followers a day (along with so many emails) since January. However 10 days ago, it changed. I didn’t receive one e-mail per follower anymore. It started summarizing it. If it is a person with a large following, Twitter sends a more personalised mail. Here are examples:

Each day, I get a summarized one like this.

By Twitter’s metrics, they sometimes send me one like this with only one notification.

@tweeklyfm gets about 20-50 followers each day, but it hadn’t changed until yesterday. Now it also receives a summarized one.

It’s a welcoming change. I wasn’t checking any of the notification mails. This sort of me brings to another point. Something that I’ve really liked about Twitter is the fact that it doesn’t do any feed filtering (as opposed to Facebook or G+). Although it comes with more noise, I like that it is objective. A public conversation can’t be had without having everyone’s voice be heard. However, Twitter might start losing the battle if more people start finding more value in other (algorithmically filtered) stream/feeds.

A great balance between the two is their searching for example: having “top” items and “all” items available to see.

What do you think? Would you be okay if Twitter started filtering their feeds?

How do you use Twitter?

I’ve been using Twitter for quite a while and my use of it has been evolving along the way. It remains one of my favourite destinations on the web (if not my most favourite).

The way people use technology is always fascinating, especially with the sudden rise in popularity of social networks the previous 5-10 years. This passionate interest has led me to where I am now: doing Masters research on how information overload affects users on social networks (specifically Twitter).

So. I suspect. Most people start using Twitter without resorting to using lists. There’s little incentive to categorize, when in the beginning the effort outweighs the benefits. That’s what I did.

At a point, it became too much and I thought I’d start putting people into lists. Based on the research I’ve read, this point is close to 150 [1][2]. If you want, you can extend the results of that to research done by people like Robert Dunbar (although that’s a contentious point).

So. Currently, I have 3 main lists. Stellenbosch, SA Web People and Web. I have my own criteria for each, and it is pretty vague. These lists I actively check and for the most part (90% of the time), I don’t miss tweets. It’s enough to not be an issue.

It’s tough for me to unfollow people. I have what you might call internet-fomo. I’d rather sit through noise and possibly find a great tweet, than unfollow people. That threshold is different to each person. On the other hand, I have 1500 followers. This home feed ‘scrolls’ through, occasionally checking it now and again, but not actively. If miss updates, I miss it. I don’t bother. I mainly use Hootsuite (although it has irritated me) as my web client, to allow me to do this.

I have a problem though. My lists are getting bigger themselves… and the people in the lists aren’t tweeting quality content. I have started unfollowing people in these lists, and it is working in that regard. However, I’ve had an idea in the back of my head for quite a while now (a year or 2), that will make the consumption of Twitter easier once people start using lists.

Basically: Based on your followers, you specify a keyword and it builds a list of 150 people. You can then specify whether it should automatically update and change. Sort of like a dashboard of graph/list management for your Twitter. I’ve manage to profile users quite well, although it works only for people who are quite active. Still have to work on that.

But yes. It’s on my bucket list of projects-to-do. Eventually. I think necessity will lead me to it. I really do enjoy Twitter.

How do you use Twitter?

—-

[1] - Validation of Dunbar’s Number in Twitter Conversations (http://arxiv.org/abs/1105.5170)

[2] - Social features of online networks: The strength of intermediary ties in online social media. (http://dx.plos.org/10.1371/journal.pone.0029358)

How NOT to scale: Tweekly.fm in 2009.

Back in January of 2009, it was university holidays and I just finished my first year of university (studying BComm Computer Science). Being kinda bored and wanting to make something, I decided to whip up a mashup called Tweekly.fm (then childishly called T.W.A.T. (Top Weekly Artist Tweet)).

It was simple. I wanted to learn how to use APIs, and further some of my programming skills. It takes your most listened to artists for the week (from Last.fm) and posts it to Twitter on a Sunday. This was an early version (circa March 2009).

Before I explain the rollercoaster ride, I want to explain my skills at that point:

- Java (in highschool)

- C (1st-year computer science)

- Game Maker

- Basic PHP/MySQL/HTML/CSS (learned it myself in high-school)

I had no sysadmin experience (no idea how to work with linode or aws whatsoever) and didn’t know of MVC frameworks. I just sort of winged it.

The nature of Tweekly.fm meant it could only go viral. Each week, users sent out tweets, and more users signed up. The week after that, more people signed up and so forth.

Now. For the most part of 2009, everything ran smoothly. Tweekly.fm was hosted on my shared hosting provider’s sites and it was performing within the imposed limits. It wasn’t hitting the bandwidth cap and so forth.

The biggest problem however (with my lack of knowledge), was that on Sunday morning 00:00 GMT+2, a CRON job called a PHP script that collected all the data from the DB and queried each user’s last.fm profile, and sent a tweet. A bottleneck quickly formed: the API calls was slowing the script down… a lot. I didn’t care really. I thought this is how these things worked. 1 hour… 2 hours… 3 hours… and so it grew.

Then I thought, considering that the tweets are sending at random times, I’m going to push the CRON back a bit, so in the middle of the 4 hour script, it would be 00:00. Cool.

Then the madness started.

My shared hosting provider sent me an e-mail:

Your website has normally used to many resources after hours and we’ve let it slip until now but now during peak times you’ve used more resources than allowed and that’s why we were forced to stop the script from running.

Basically it seems like your opening lots of connections with your script instead of just running one script doing everything and that’s what is causing the extra loadon the server.

Being in the middle of a uni term, I didn’t have time to sort it out (my provider kept mailing me). No idea what to do, I started searching for better servers (with no idea what I was looking for). I had no money too. So I started a donation drive amongst the users to fund it. I got enough (thanks! you know who you are) to get a decent server going for 2-3 months (I didn’t think past that moment).

I started with Media Temple. I think was the grid service option. Took me 2 days to transfer everything and getting it up and running. At that stage, I didn’t have my own DNS nameserver, so I had to point it to the servers each time. It took about two days to resolve, so I had to “run” two websites simultaneously (the old one just said we are moving, and the new one actually worked). Also: Moving servers = dumping the db into a 4mb SQL script (through PHPMyAdmin) and uploading all the files through FTP. I knew at that point there had to be an easier way, I just didn’t know about it.

It was up on Media Temple, and I was ready to see what was going to happen. Giving up my Saturday night, I sat by my PC…

…and then the server kept crashing.

It was running out of virtual memory extremely quickly shutting the server down. It then repowered up by itself. But: Now only a few tweets were sent. So I search Twitter, checked who the last user was, found his position in the db, and ran the script again changing the index pointer (mysql_data_seek) of the mysql result set… and then it crashed again. I continued for an hour, fell asleep, and finished up the rest the next day. I can’t do it again like this, I had to find another way… within a week, while doing uni work. So, I moved back to my shared hosting. I was hoping they just won’t notice.

That week our family was off the visit my sister in the Karoo, so that was my only option.

So there I was in the middle of the Karoo, having a break around the braai, drinking some brandy and enjoying my time with the family when I realised I forgot to take out the “mysql_data_seek” line from the code…

Now, the Karoo is big and in some places you don’t have phone reception. It also gets very cold at night. It was akin to this: except night.

So I called my friend. I had 30min left before the script would start. In the nick of time, I gave him instructions to FTP in and edit out the line. SUCCESS.

It was short-lived though. When I came back I had to move servers again. This time I opted for Rackspace’s cloud service. I did more research this time of course. It scales automatically, so by definition, the memory wouldn’t run out, right? Cool. Let’s go.

Come Saturday night again, I sit and stare at the screen… Everything is going smoothly… until 15 minutes in the script just stops. FFFUUUU. There was a hard-limit on how long scripts could run on rackspace. I gave up. I had no idea how to handle this. My skills weren’t up to scratch and it was 1am. I felt hopeless, shed a tear and went to sleep.

…

The next week, I stopped the service, while I thought what to do. I tried selling Tweekly.fm to Last.fm, talking to CBS over the phone, but no luck. So I put it back on my shared hosting and made it manual. The final nail in the coffin of a fun experiment… until a few weeks later, Scott (@dordotky) started a similar service. I mailed him, talked some terms out, and we have been working together since on it. He has been a tremendous help in scaling the service since then. We don’t have much time to work on it (both doing other things full-time), but at least it is still going and running smoothly (not crashing servers).

If you want to scale today. Don’t do what I did. Ask people to help (to those that did, thanks!).

What did I learn from the whole experience?

1) Don’t use shared hosting for anything more than a simple static site. This advice might seem old now, but hey, I didn’t know of any better. Nowadays, however, there are much much more cloud services available that is easier and better and cheaper.

2) If something seems overly complicated, it probably is. While I was working on a weekly schedule, it would’ve been better to forego a week’s tweets to take time and learn and research the better options! I should’ve loaded up my own DNS server (managed or otherwise), I should’ve learned about dvcs so that I don’t have to use slow FTP to upload files, etc.

3) Most of all however. If you have no idea what you are doing, just wing it anyway.

TwimeMachine v3.0

I’ve had quite fun developing the new version of TwimeMachine. Twitter is deprecating the ‘page’ parameter, so I took a different approach. See the blog post.

Django, is still the back-end, but I used a lot of javascript/jquery for the new version. It is essentially a one-page app now. I’ve thrown in search now (because it loads the whole set of tweets). From a design perspective, it was quite an interesting exercise. I tried to keep the lag down as much as I can while the tweets load so as to not make it seem jittery. Not a great UX. At first it was quite a mission to get the ‘clock’ on the left working so that it corresponds to the opposite tweet. I eventually got a work-around going by storing the positions of each tweet, and then on scroll gets the window’s position, and checks whether a tweet is there (in the array). It is still not as fast as I would like. Storing all the positions is the bottleneck and most expensive part, and causes the millisecond “lag”. It is much better than it was though. This also meant that I couldn’t have any loading/spinning gifs, because that millisecond makes it jump.

Check it out, let me know what you think!

Stuff I’ve said.

I’ve said some stuff in the past. Some of it, I am proud of. Since I’ve started with this new blog, I thought I’d make a recap of things I’ve said.

I write occasionally for Bandwidthblog (I’m due for another post). Here are some of it:

Why Facebook and Spotify are on the right track.

I’ve been in the automatic sharing game for a while with Tweekly.fm. I talk about why Facebook and Spotify’s approach is going to work.

The Future of Interfaces: Why OS X Lion looks like iOS.

To put it simply. The iPhone and iPad are their highest selling products. Apple wants to move their users to the rest of their ecosystem.

Should Wikipedia’s Search be improved?

Based on a competition by Greplin, I pondered whether Wikipedia’s search should be improved. I talk about the nature of knowledge and why people use Wikipedia and how they get access to the information they need.

Should I use Google+?

I ponder Google+’s sharing mechanism.

Here are some posts from my old blog:

Soup, Dunbar’s Number and Facebook.

I am fascinated with the psychology of groups, especially how it relates to online relationships. I ponder Dunbar’s Number and its effects.

Facebook login fail and semantics.

There was an interesting event in early 2010. RWW ranked higher on Google for the term “Facebook login”. We all laughed, but it isn’t actually that funny. We have to start thinking differently about user experience now that the rest of the world is coming online.

———-

Hopefully I’ll be saying more things.