blog.bjones.net

I’ve been using Tweetnest to download and archive my tweets for some time now. I set it up some time ago, and generally, it just runs. I think I’ve had to go reboot the cronjob it uses to collect new tweets once or twice. I just added a feature to automate finding tweets “on this day” from prior years.

Background

Tweetnest, if you’re unfamiliar, is a self-hosted app to collect all your tweets and present them nicely in a way that’s separate from Twitter’s interface. It collects everything into a mySQL database, and is written in PHP. It’s nice to have a backup of Tweets kept separate from Twitter.

The main page just displays recent tweets, but it also lets you search, and has month and day pages. It’s a nice, calm interface for exploring or searching old Tweets of yours, which comes in more handy than you’d think.

I have been using Micro.blog more, through this site, which solves some of those problems. But even if I left Twitter today, having this backup and nice interface to it would be valuable, so I figured it was worth doing a little writeup on.

Ok, so what’s the new feature?

As I said I haven’t really touched the site since installing it, and mostly it works fine. But I recently went in a made a tweak, which is the impetus for this post.

One thing I’ve been doing recently is checking out my Tweets “on this day” to see what was going on last year, five years ago, even a decade ago. Twitter doesn’t have a function for this natively, and neither does Tweetnest.

I started out just by visiting the pages on Tweetnest sequentially. The daily page URLs are formatted year/month/day, so you can visit last year’s pretty easily, then just change the year in the URL, see the new ones, and so on. But this is tedious and occasionally there are no Tweets on a given day so it’s a waste of time.

I quickly tired of that, and wrote a little Keyboard Maestro macro to get the current date, format the URL properly, and open tabs in my browser, one for each year between one year ago and when my archive starts in 2008. This is pretty good, but it only works on desktop, not mobile, and it still suffers from the case when there are no tweets on a day (it just opens a tab that says “No tweets here!”).

Finally, I decided to create a proper solution. I built a page for the site that gets the current day and month, then queries the database for tweets that happened today, and presents them.

The code

This was pretty simple and required no innovation on my part. I just modified the code for a specific day to remove the year, and added logic to get the date from the current date, instead of the URL. It’s functional, but since I don’t know PHP it’s quite possible I’ve done something wrong. Here’s the query:

date_default_timezone_set('America/Chicago');
$day = date("d");
$month = date("m");

...

query("SELECT `".DTP."tweets`.*, `".DTP."tweetusers`.`screenname`, `".DTP."tweetusers`.`realname`, `".DTP."tweetusers`.`profileimage` FROM `".DTP."tweets` LEFT JOIN `".DTP."tweetusers` ON `".DTP."tweets`.`userid` = `".DTP."tweetusers`.`userid` WHERE MONTH(FROM_UNIXTIME(`time`" . DB_OFFSET . ")) = '" . s($month) . "' AND DAY(FROM_UNIXTIME(`time`" . DB_OFFSET . ")) = '" . s($day) . "' AND `".DTP."tweets`.`hidden` = 0 ORDER BY `".DTP."tweets`.`time` DESC");&lt;br&gt;

Next, I modified the .htaccess file to run the new php file at a particular URL (I called mine “/today”), and the inc/html.php file to include a link to “/today” in the sidebar. And that’s that. Now I can visit that link and see all the tweets posted through the years on a single page. You can see my modifications on my Github fork of the project.

Other enhancements

One other thing about Tweetnest is that it hasn’t been officially updated to include the new, longer Tweets. So some folks who use it have modified it to take advantage of those. You can see those changes in my previous commit, or in various issues in the main repository.

One shortcoming

A thing I noticed while working on this was that, because I hadn’t made the long-tweets upgrade until now, many prior tweets are truncated (Twitter increased the length at the end of 2017).

One suggestion in the Github issues was to remove all tweets since the lengthening and re-import them.

Tweetnest has two ways of importing tweets: The loadtweets.php file and the loadarchive.php file.

Loadtweets is the thing that runs on a cronjob, so usually it’s only grabbing a handful of tweets at a time. It uses Twitter’s API to get tweets, and so is limited to 3,200. This, unfortunately, isn’t quite far enough to get all my Tweets since the change (it also hits a memory error when trying to load that many).

Loadarchive was built to consume json files you can download from Twitter that contain your archive. Twitter limits public access to tweets to the last 3,200, but it will compile your data and let you download all of your tweets.

Unfortunately, the format of this archive has changed since this function was written for Tweetnest. I attempted this route, but ran into a couple roadblocks:

The link to the tweet on Twitter is missing its username. Tweetnest generates the Tweet, but wherever it’s trying to grab the username from to generate the link is missing, and so the URL it links to is just missing a piece.
Retweets are messed up. In prior archives (and in current API calls) you seem to get much more information about retweets than you do in the archive. And so when you load tweets from the archive, Retweets aren’t able to get the original tweet.

There are probably ways around both of these problems for someone more versed in PHP and/or Twitter’s API/data than I am, but since new tweets should come in at their full length and new retweets are ok, it’s not vital to fix these problems. (If you do tackle this, please let me know).

Conclusion

Tweetnest is pretty great, especially for some free software written nearly a decade ago. It’s nice to be able to hack on things and add features that make your life slightly easier.