Introduction

Imagine being able to control your Nest thermostat and unlock your car through Siri. With GoogolPlex, these are now possible.

We are Ajay Patel, Alex Sands, Ben Hsu, and Gagan Gupta, four freshmen in the M&T program at the University of Pennsylvania. Our project took 3rd place overall at PennApps (the largest university hackathon sponsored by companies like Google, Facebook, and Dropbox) out of over 220 projects. We’re writing this article to explain the technical aspects behind GoogolPlex. At the end of the article, we explain how you can install this on your iOS device and try out some proof of concept commands.

GoogolPlex is a hack on Siri (iOS 7 compatible) that allows Siri to integrate with third-party apps and hardware with custom commands like “turn on the lights.” Setup takes a minute and no jailbreaking is required. GoogolPlex already allows you to play songs in Spotify, pay friends through Venmo, Instagram selfies, and control Philips Hue lights with simple voice commands, all through Siri.

And what we’re even more excited about is that, in the future, third-party apps can integrate with GoogolPlex themselves and allow for their functionality to be accessed through Siri.

SiriProxy

Prior to our hack, there was a similar solution that existed called SiriProxy, which is as of now defunct. SiriProxy relied on intercepting network requests within Siri’s internal network. The problem with this is that it gave Apple an easy way to shut it down because Apple controls how its own internal network protocol requests are made and verified. Sure enough, in the iOS 7 update, Apple cutoff support for SiriProxy:

“SiriProxy does not (at this time) work with iOS 7. Significant changes made to the Siri protocol stack have rendered SiriProxy inoperable with iOS 7. … However, SiriProxy may never support iOS 7.”

In addition, SiriProxy required a great deal of setup. SiriProxy’s setup involved various complicated steps that required the user to mess with terminal commands and setup their own proxy server.

GoogolPlex, in addition to being much simpler, does not face this problem.

How We Hacked Siri

We named our hack “GoogolPlex,” to symbolize the near-infinite number of possibilities Siri could reach with our hack. But there is also a strategic and necessary reason we named it this. When a user says “GoogolPlex, turn off the lights,” Siri hears this as “Google ‘Plex turn off the lights,’” interpreting that the user wants to Google something. Siri then searches Google for the words “Plex turn off the lights,” and this is where GoogolPlex comes in. We are able to get the words the user was asking Siri by intercepting the request between Siri and Google with a man-in-the-middle exploit.

How DNS Spoofing Works

With a custom built proxy server, we make the phone connect to a custom DNS server. The DNS server system is essentially the system by which human-readable addresses like “google.com” are converted to IP addresses like “127.28.251.124.” By connecting the phone to our custom DNS server, we point “google.com” to our own server, redirecting requests to “google.com” to our server.

Putting it Together

Putting the vulnerability we found together with a proxy server that spoofs the DNS of “google.com,” we are able to send the string of words the user asked Siri to our server. Then, we can perform custom actions using third-party APIs to interact with various services and hardware.

Performing Actions

After receiving a string of commands like “Plex turn off the lights,” our server uses natural language processing methods to understand the command. If the user said “Plex tell me a programming joke,” and the appropriate response is text, we load a webpage with the response.

However, if the appropriate response is an action that requires interacting with an external API like Spotify, Venmo, Instagram, or Philips Hue, the webserver sends a HTTP Redirect Response to the phone with a custom URL scheme like “googolplex://.” This URL scheme then launches the GoogolPlex helper app installed on the phone that connects to the appropriate API and performs the custom action.

Intelligent Proxy Connection and Scalability

As stated above, we use a remote server to proxy the connection between the phone and Siri to introduce a custom DNS server that spoofs “google.com.” In order to reduce load on our remote server and maintain the privacy of users, we only proxy requests from Siri that explicitly begin with “Google Plex.” This is done with a client-side decision making script that reads the request being made and decides whether or not to send the connection through the proxy server. Since the server then only deals with a few requests, and is only proxying a marginal amount of network traffic (kilobytes/a few megabytes) per user, it can easily scale as it is built on EC2 instances. Your network traffic only goes through our proxy server if you explicitly ask Siri for “GoogolPlex.” The final result is privacy for our users and lower server costs for us.

One-minute Installation

If you want to try this on your iOS Device, go to http://www.betterthansiri.com. Installation takes less than a minute and works on ANY Siri-enabled iOS device. However, since the GoogolPlex app is not available on the App Store as of now, you cannot perform actions that require the helper app, such as Spotify or Hue. However, you can try some proof-of-concept commands like “GoogolPlex tell me a programming joke.”

Where This is Headed

We plan on working on GoogolPlex in the near future to make it open for anyone to build custom commands and actions into the platform. For now, use the instructions above to try some proof-of-concept commands. While this hack demonstrates an awesome use of technology to get around restrictions, the only true way to get a great experience with third-party apps in Siri is if Apple builds in an open API.

Contact

To contact us with any inquiries/comments about GoogolPlex or this project email patel.ajay285@gmail.com.