This is a write up for day four out of five. here’s a link to the other days:
note: I’ve learned never to promise someone the next post in a series is coming soon. I did this with the last post, and then waited over a month before posting this up! oops
I had started to go a little delirious from the bites and lack of sleep, but this was it, my last stand. I cornered the hotel staff and convinced them to let me keep the canister of mosquite repellent in my room throughout the night. A few hours before bed, in between coming back to the hotel and having the catch up with the team, I had made sure to spray every corner of the room. When I returned after dinner a few hours later, I sprayed the room again. I got ready for bed. Another round of spraying. This was war, and I was determined to get at least 1 nights sleep.
I awoke, seven glorious, sleep filled hours later. Success!!
The only thing on the agenda for today was the long drive back to Kampala. It had been a long week -an amazing week, but a long one and I was looking forward to getting back home. I get ready and pack my bags. On my way out I triumphantly place the now empty canister of bug repellent in the bin and head across for breakfast.
Cary arrives and has a troubled look on his face. He has a backup android phone with him that he was using to test out the deployment on, and when he tried to play around with the app last night, it wasn’t quite working. He also tried to install rapidFTR onto his personal mobile, and it seemed to be working fine there, so we were a little confused as to what was happening. After a bit of thinking and playing around with the phones, we decide that maybe it’s just something quirky with the backup phone. We weren’t as careful and organised with the set up of that phone and decide that maybe during the various dummy deployments and tests something had gone wrong, after all the app was working perfectly fine on Cary’s own phone.
We finish our breakfast, and get into the cars, ready for that long drive back. as we start to drive, theres a nagging feeling that perhaps something is actually wrong, and we debate whether we should quickly stop by the camp on our way home to check everyones phones. Now it just so happens that our route back passes the Nyankabande transit center. We almost decide to drive straight past it, but then at the last second Cary calls Zubeida to stop the car, and we turn around and drive into the camp.
We arrived at the camp a while before most of the volunteers that had the phones. it was a bit of a frustrating wait, we knew the longer we had to wait the longer our journey home became, and if you’ve ever experienced the Kampala traffic you’d understand why we were keen to get back at a decent hour.
It takes a while but eventually everyone turns up. I’m pretty sure you can all guess by now what happens. The issue we were seeing with the phone was not a one-off, and every one of the android phones that we had deployed had stopped working! it’s one of those “oh crap!” moments, but at the same time how lucky is it that we were still around to witness the issue and attempt to fix it.
So there we are, out at the camp, Cary with the phones checking they all have the same issues, me on my laptop trying to tether the data connection from my phone to see if I can see anything dodgy on the production servers. But before I get too far into it, a realisation strikes. While it’s every developers natural tendency to try to understand and solve a problem they see, now is not the time to do so. The phones are clearly not working, and the ‘why’ is not as important as getting it fixed. I stop what I’m doing and we take a call to wipe the phones and reinstall the APKs on each of them. We had already tried this with the test phone and it had seemed to fix the issue. I had an inkling what the issue was and stored that thought to one side, planning to explore once we were back on the road.
We finish getting the phones set up, test them all out and all was well, disaster averted. It was a pretty crazy couple of hours, but was a great learning experience, and a great story to tell
We get into the car, say our goodbyes one last time and drive off. As soon as we get into the car I open up my laptop, after some exploration of the code I think i figure out what the issue is, but by now we’re driving through the mountains and have no internet connectivity so I can’t prove my theory. It turns out that this backup server (mentioned in the previous post, a duplicate of our production server that couch replicates to every 10 minutes) was not getting a complete replication. When a rapidFTR mobile application authenticates itself to a server for the first time, we generate a unique encryption key that is used to encrypt the data on the phone. This key is stored on the server and is sent every time you successfully log in, and is used to unlock the data on the device. However the”db-key” database was not being replication/backed up. So when we rebuilt and redeployed the server, and restored from the backup we lost the encryption keys.
This meant that when the volunteers tried to log in the next morning, the server didn’t have an encryption key on record for the device’s IMEI, and so assumed this was a new device connecting for the first time and generated a new key. Of course this new key was incorrect and the phone was unable to unlock the encrypted data and the application crashed.
In attempting to figure this out I had been staring intently at my laptop screen while we drove along some pretty bendy roads around the hills. When i finally looked up, I suddenly realised what a bad idea that was. I then spent the remainder of the journey with the window rolled down trying my level best to recover from the most intense motion sickness I had experienced to date, and trying not to throw up.
You’ll be glad (I hope) to hear that I managed to survive the journey without throwing up. We completed the rest of the day without any other major events.
So that’s it, this is the sum total of my experiences at Nyakabande and the first ever deployment of RapidFTR.
Since then we’ve moved ahead quite significantly, I’ve been involved in an inception for the next phase of work and the project is looking like it’s going to move into some pretty interesting new areas. As always, to keep up to date with the latest of what’s going on with the project hop on over to the official google group
, and checkout us out on github