On-device testing and continuous deployment for mobile game development

Disclaimer: This is a technical post. If you are not an engineer or if you have no preferences for continuous integration, please stop reading here 😉

Better quality in iterative game development

Game development is a highly iterative process and, therefore, very complex and prone to bugs. To improve the overall quality of our software, we try to automate as many software development steps as possible. Compilation, on-device testing and deployment to alpha & beta testers are core tasks that need to be performed ideally after every single commit to the project source repository (Mercurial SCM in our case).

This post is about our goal to enable game developers to have their apps tested as often as possible and about the tool chain setup to bridge everything between development and before releasing an app on different stores.

Continuous Integration infrastructure & setup

At Gbanga, we use an internet connected Mac mini with Jenkins installation as the basic build infrastructure. Development tools such as Android SDK, Xcode, command line tools, several versions of Unity3D, Mercurial client, Fastlane tools and PhoneGap’s ios-deploy are installed locally. A set of hardware is attached to a powerful USB hub (make sure you have one that has enough ampere at the output plugs; we are happy with the Exsys USB 3.0 metal HUB, 7 ports model).

In Jenkins, we have installed plugins for Mercurial, HockeyApp, Unity3D and Slack Notification. The Jenkins job pipeline is organized as follows (identical for ios):

1. MyProject-build-android
2. MyProject-deploy-android-local
3. MyProject-deploy-android-ha
4. MyProject-deploy-android-alpha (or -beta)
5. MyProject-build-android-release
6. MyProject-deploy-android-release

1. Building from sources

Firstly, the game project is fetched from the Mercurial repository at Bitbucket. The Jenkins job accesses Bitbucket through SSH and the job is triggered with a web hook and an authentication token that you get in the settings of your Bitbucket project (add a POST address under Services). The project from the source is then opened in the correct version of Unity. A command line parameter forces Unity to run a build script after successfully opening the project. The artefact is then stored as an APK (for Android; or IPA for iOS).

2. Deploy to connected devices

Secondly, the artefact from the previous job (the APK or IPA) gets deployed to all attached devices. We use the following shell script that directly interacts with the Android SDK’s command line tool adb that iterates over all connected devices and installs the APK (or IPA) on it:

FILEPATH="Build"
PACKAGE="com.gbanga.mygame"
ACTIVITYNAME="com.unity3d.player.UnityPlayerNativeActivity"
APK=`ls $FILEPATH/*.apk`
installOnSingleDevice(){
echo "Installing on $DEVICE"
adb -s $1 install -r $APK
adb -s $1 shell am start -n $2/$3
}
if [ $? -eq 0 ] ;then
adb devices | grep device | grep -v attached | cut -f 1 > tmpfile
while read line;
do
installOnSingleDevice $line $PACKAGE $ACTIVITYNAME
done < tmpfile
rm tmpfile
else
echo "No apk at this path (${FILEPATH})"
fi

For iOS, we use PhoneGap’s ios-deploy and the following command lines to get all connected devices:

system_profiler SPUSBDataType | sed -n -E -e '/(iPhone|iPad|iPod)/,/Serial/s/ *Serial Number: *(.+)/\1/p' > tmpfile

/usr/local/bin/ios-deploy --bundle $APPFILE --verbose --timeout 10 --id $1 || true

Piping the ios-deploy command to || true is handy to have the app installed on as many devices as possible. Of course, this is against the fundamental meaning of a successful job in Jenkins; however, for us it is a best-effort approach. There are several situations where installations fail: wrong provisioning profiles, locked screens, empty batteries, etc. We don’t care in all those cases. Further, you can easily have a look at the output of the particular job later to solve such problems.

3. Deploy to HockeyApp

Every day at 7pm (H 19 * * *), Jenkins deploys the app to our HockeyApp account for alpha and beta-testing deployment. All participants are notified with an email. To prevent endless uploads and email notifications for projects that have been untouched and unchanged, we use flexible publish with a conditional action based on the following shell script:

/usr/bin/GetFileInfo -m output/*.ipa
MDATE=`/usr/bin/GetFileInfo -m output/*.ipa | cut -d' ' -f1`
DATE=`date "+%m/%d/%Y"`
if [ "$MDATE" != "$DATE" ] then
exit 1
fi

4. Deploy to app stores

For Continuous Deployment (the new hipster on CI block) to app stores (both, Android and iOS), we use several tools and have put together our individual tool chain.

For iOS, we use the great Fastlane tools. Since Unity3D creates the Xcode project from scratch, we need to inject the Fastlane folders manually. For that, we initialize Fastlane with the fastlane init command in the generated Xcode project of the very first build-ios job. After that, a job runs the shell script:

fastlane ios beta

The Fastlane file has the following pipeline configuration:

sigh
gym
pilot

And the Appfile has the following variables set:

app_identifier "com.gbanga.mygame"
apple_id "ourcompany"
team_id "1980JU161515"

This configuration sets up an automated, non-interactive execution that uploads a build to Apple’s TestFlight system. However, builds are processing for quite a while (several hours or several days). Like that, the Jenkins job has to wait for very long to assign alpha/beta testers and to release it.

5. & 6. Toolchain for release versions

The only difference between the release jobs 5 and 6 and development jobs 1 and 4 is the branch in Bitbucket that is used to compile. So, a release manager or developer merges back the development branch to the release branch and triggers job 5 and 6 to run and deploy to the app stores for review.

Summary

By installing a few standard open-source projects, configuring Jenkins, adding customized shell scripts and attaching hardware, we lowered the pain for developers to test their game apps frequently on a hourly and daily base. Daily deployments to a broader group of testers happen automated through HockeyApp and make our game software more accessible to all involved parties.

Possible improvements include the integration of on-device monkey tests and the automated generation of screenshots.

In a future blog post, we will discuss development with Unity3D on a source control tool like Mercurial and the challenges involved in concurrent editing of scenes and assets.