Personal Containers

Towards Statistical Queries over Distributed Data

Amir Chaudhry — Thu, 26 Jan 2012 00:00:00 PST

One of the issues with the distributed nature of personal data is the ability to perform statistical queries over it. This is true of both aggregated data stores and those that only catalog data, which remains in its original location. In both cases, a client that wishes to run a query still has to request access to a number of stores and aggregate the results in an efficient manner. This last piece is exactly the scenario for which Dataware is being designed.

A new project is taking shape, which will use personal containers and the principles of Dataware to build an employee incentive scheme to optimise energy usage. It’s taking place at Cambridge University in two phases. The first phase is to gather fine-grained data from employees regarding travel habits and energy usage. The second phase aims to use this information to construct incentive schemes to help optimise the energy usage of employees and monitor the changes.

Energy use in buildings and in transport systems is rapidly becoming part of our lifelong contextual footprint so the study needs to be completed without participants sacrificing their privacy. Therefore, the project will enable a user to collect their private information into a personal container which they own. Then the project team can request the ability to run queries over subsets of the data contained in those stores. This arrangement facilitates the following:-

Allows the research team to calculate energy use accurately
Assures users that the data is only used for this application
Denies the team access to other data that is not relevant

The type of sensitive data relevant to the study will be daily commutes (e.g. GPS information between 8am-10am and 4pm-6pm on weekdays only). This information could be combined with public data (e.g. traffic data) and a series of incentives constructed to optimise the overall energy usage. A particularly exciting possibility would be the ability to test various types of incentives with different groups to see which are most effective. All while preserving the individual privacy of users.

We’ll post more information as the project develops, including the hurdles and technical challenges that need to be overcome. In the meantime, you can read more about the project itself at the C-AWARE site.

Privacy needs to be built in

Amir Chaudhry — Wed, 25 Jan 2012 00:00:00 PST

Today, news broke about how O2, a Mobile Network Operator, reveals customers’ mobile numbers when they browse from their phones. A site put together by Lewis Peckover simply displays the headers sent to his servers when people visit the page. If you visit from an O2 phone, then you ~~can~~ could easily see your phone number displayed. In other words, the phone number is simply given to the requesting server as plain text in the http headers (specifically x-up-calling-line-id). There was nothing particularly special about Lewis’ site so it’s reasonable to assume that any site people have browsed using their 3G connection could have kept a copy of the user’s number.

This is obviously a massive breach of privacy and as you can imagine, news of it spread fairly quickly via twitter. A few customers even posted templates to help others file complaints with the UK’s Data Protection office (who have now contacted O2 to “remind them of their data breach notification obligations”). One of the main objections was that advertisers who can track visitors now also have access to phone numbers which can be used to tie different visits together.

In fact, this isn’t the first time that mobile browsing has been shown to leak data. A prior study by Collin Mulliner in 2010 showed that the phone number could appear in any number of headers as well as other information (e.g your roaming status).

This highlights how ‘spread-out’ personal data has become and breaches like this serve to remind us how lax some organisations can be. As more and more people access the web via mobile devices, issues like this become more important. Products and services should be designed to be privacy-preserving from the very beginning. To some extent this can mitigate any mistakes that others can make but can also empower users by giving them the option to explicitly share information if they deem it sensitive.

(O2 has now written a post about what happened)

Growing awareness of Personal Data

Amir Chaudhry — Fri, 20 Jan 2012 00:00:00 PST

Awareness of Personal Data issues have risen in prominence in recent years and there have even been a number of new companies springing up around the idea of personal data stores. At the moment, only a handful of these are public and it’s likely that more of them are working away behind the scenes.

Although we won’t necessarily be discussing these sites in any detail it’s interesting that what they all have in common is the aim of aggregating a user’s data in one place. Once a user has collected all her data, the suggestion is that she can share those ‘data-streams’ with others, including services that she’s granted access to. At the moment there’s limited information on how this aspect would work in practice but it’s certainly important in order to liberate the value of that data.

In general, this approach seems similar to the previous iteration of Personal Containers. The idea of aggregating data into one place is fairly intuitive but as we’ve learned from prior feedback, there is valid concern about having “all your eggs in one basket”.

There have been security issues that highlight this problem. Last year a programming error at Dropbox, a popular file-syncing service, led to customer accounts being accessible. For a few hours you could access any user account with any arbitrary password. The flaw was possible because of the way Dropbox handles encryption, specifically that they hold users’ encryption keys and therefore have access to files.*

The situation above is a clear example that there should be more separation of privileges among our data. Our response to feedback was the concept of Dataware. The premise is to leave the data themselves in the most appropriate place but facilitate access to them. We achieve this by creating shims that can interface with services and at the moment the team has put together a couple that interface with social networking sites and another for Aether’s Notebook. These look very promising and we hope to continue developing these and others, along with the ability to run authorised computations on the data.

There’ll be more to follow in the coming weeks on different aspects of Personal Containers, covering both Dataware and Signpost.

*Although this might sound odd, many services do this. If a user forgets their password, the files can still be recovered. In this case there was some confusion about how security was presented to users. In order to have a fully secure system, users would need to accept that if they lost their passwords, the service wouldn’t be able to help.

Aether's Notebook

Richard Mortier — Thu, 01 Sep 2011 00:00:00 PDT

Aether’s Notebook has been developed by Dominic Price as part of the Horizon Infrastructure project. For more details see the source or visit its wiki.

Many applications that make use of personal data start by logging data on mobile devices before subsequent upload and/or offline processing. Aether’s Notebook is a framework that separates the concerns of recording and storing log data. Structured as a client-server framework, it makes logging similar data by multiple applications efficient, provides a single interface for the user to control what is logged, and makes it straightforward to log new data types in a consistent fashion.

The cient component consists of loggers which generate log messages, appenders which consume them, and a core which bridges messages between loggers and appenders, replicating them as required. The core can also be made responsible for starting/stopping selected managed loggers and appenders. Log messages contain, in addition to the messages, three pieces of metadata: timestamp including timezone; location as given by the best available source, and a UID and version number for the logger. Location logging can be turned on and off.

There are several built-in loggers: cell location, connection state, signal strength, and position; and two built-in appenders, one to a local (on-device) file and the other which will take local log files and HTTP POST them to an external server.

Aether’s Notebook is now available in the Android Marketplace. To download the AGPLv3 licensed source, visit source. For more detailed, live documentation visit the wiki.

Moving Personal Containers to the next stage

Richard Mortier — Wed, 31 Aug 2011 00:00:00 PDT

After some time analysing responses to the first prototypes of the Personal Container, we’re now moving on to the next stage! One of the pieces of feedback we received quite strongly was that it’s not always appropriate to attempt to pull all your personal data into a single centralised location. This seems to be true whether or not you make that location locally hosted, e.g., on your phone or set-top box; or you make that location cloud hosted, e.g., in an Amazon virtual machine or in a specifically designed Personal Container service. A number of people we talked with expressed high levels of concern about the security of such a repository, its vulnerability to attack and the fear of the “all your eggs in one basket” approach.

We had a think about this and, in response, came up with the notion of Dataware. This lets you construct a federated system in which you control access to your personal data, which remains stored where it is most appropriate. For example, your financial data might remain with your bank, and your health data with your GP; but you would have the ability to permit access to these data sources by third parties, for specific, restricted purposes. Equally, where appropriate, you might construct aggregate data sources that live on hosts such as your set-top box or phone.

Some of the technical problems encountered in trying to build this system are described in overview on this website. Over the coming weeks we will be adding a series of posts to this blog describing some of them in detail, along with some of the components we’ve been building, with the hope that our experiences and our code will be of wider interest and use. We look forward to any comments or other inputs!

Project VRM

Tue, 30 Aug 2011 00:00:00 PDT

“ProjectVRM seeks to improve markets by equipping customers with tools for both independence from vendors and better engagement with vendors.”

Internet Identity Commons

Tue, 30 Aug 2011 00:00:00 PDT

“Identity Commons is a community of groups working on developing the identity and social layer of the web. We are loosely connected sharing a common purpose and principles. Our main community gathering is the Internet Identity Workshop that happens twice a year.”

Personal Data Ecosystem Consortium

Tue, 30 Aug 2011 00:00:00 PDT

“The Consortium catalyzes a Personal Data Ecosystem where individuals control their own data by enabling a thriving network of businesses around personal data stores and services. Our three constituency initiatives:

The Startup Circle - for startups committed to putting people in control of their own data, open standards and interoperability
Industry Collaborative - for existing industries seeking to understand opportunities, launch pilot projects and ultimately offer services in the ecosystem.
1 Million People for Personal Data - gathers people enthusiastic about the vision and keen trying out new products and services.”

Dataware Networking

Wed, 13 Apr 2011 00:00:00 PDT

Keeping it personal

Wed, 23 Mar 2011 00:00:00 PDT

Computing is becoming pervasive. The development of commonly available platforms has opened ubiquitous computing to a massive and creative developer base. We see a common application template of smart phone and cloud computing service, bringing personalized experiences to the user while companies seek (often desperately!) to monetize the information derived from the use of the applications. This sharing of information and co-creation of value is at the heart of the digital economy, but implementation for ethical companies often hits a privacy brick wall, while others are blissfully unaware of the privacy minefield they are walking through. The talk will discuss some of the issues in personal information sharing, the deep social context in which this sharing takes place in the real world, present some challenges for the future in embedding this sharing in technology and some of our work in this field.

Unclouded Vision

Wed, 05 Jan 2011 00:00:00 PST

Current opinion and debate surrounding the capabilities and use of the Cloud is particularly strident. By contrast, the academic community has long pursued completely decentralised approaches to service provision. In this paper we contrast these two extremes, and propose an architecture, Droplets, that enables a controlled trade-off between the costs and benefits of each. We also provide indications of implementation technologies and two simple sample applications that substantially benefit by exploiting these trade-offs.

The Dataware Manifesto

Tue, 04 Jan 2011 00:00:00 PST

In this paper we concern ourselves with Service-Oriented Architectures (SOA) in the “business to consumer” (B2C) arena. In particular we consider the services required to enable consumers to combine data they possess with data held about them by businesses and government. We introduce the concept of Dataware as the logical federation of data sources containing “my data” and discuss an SOA to deliver new and compelling services and applications able to reap the benefits of value-in-use for consumers.

Becoming Dataware--- Enabling third-party computation across persnonal data

Thu, 16 Dec 2010 00:00:00 PST

Modern life involves each of us in the creation and management of data, and specifically digital data. Data about us is either created and managed by us (e.g., our address books, email accounts), or by others (e.g., our health records, bank transactions, loyalty card activity). Personal Containers is a project investigating how to build an ecosystem around my data, supporting provision of novel, desirable applications and services by new and existing businesses. The key technical problem in supporting an ecology around my data is not one of containment (“how can I archive all of my data?”). The matter is complicated by the basic property of digital data, that it can be infinitely copied without loss of fidelity: once my data escapes my immediate purview, I cannot easily exercise further control over it; yet in order to generate significant value from my data, I must allow others access to it. I will describe our initial steps toward a system in which we are trying to enable third parties to compute over personal data while providing individually acceptable privacy guarantees.

Horizon--- Becoming Dataware

Thu, 18 Nov 2010 00:00:00 PST

As we go about our lives, each of us creates and manages personal digital data about our online and real-world activities. Horizon Digital Economy Research is an RCUK research hub investigating the many different challenges surrounding collection and exploitation of these personal contextual footprints. Currently, many companies exploit our contextual footprints for their own gain, often without much explicit understanding or involvement on our part. Building an ecosystem around exploitation of our contextual footprints that maintains acceptable levels of privacy, both when our data is being exploited individually and as part of a group, is key to enabling growth in value of our social and personal data. After introducing Horizon, I will describe our initial steps toward a system in which we are trying to enable third parties to compute over personal data while providing individually acceptable privacy guarantees.

Mirage, A New Multi-Scale Operating System for Clouds and Crowds

Fri, 22 Oct 2010 00:00:00 PDT

Applications run on all kinds of environments these days: multicore desktops, virtual cloud infrastructures, smart-phones, and web browsers. These diverse environments make it worth rethinking the long-term future of our software stacks; do we really want to continue bundling gigabytes of general-purpose OS software with every single cloud image? Is there any point holding onto decades-old interfaces such as POSIX any more?

I will introduce Mirage, a new operating system built in the statically type-safe OCaml functional language. Mirage compiles high-level functional source code directly into a variety of targets such as:

small microkernels that run directly on the “bare-metal” Xen hypervisor;
Javascript for web browsers; or
embedded ARM devices; …and of course normal operating systems such as Linux.

Mirage provides a consistent, simple programming API across all of these diverse backends, which makes it a powerful foundation for constructing safe, complex distributed systems across a heterogeneous set of modern compute resources such as mobile devices or cloud computing infrastructure. Also, it’s just plain fun programming in OCaml.

Personal Containers or, Your Life in Bits

Mon, 11 Oct 2010 00:00:00 PDT

Unclouded Vision

Mon, 11 Oct 2010 00:00:00 PDT

Free the data

Fri, 01 Oct 2010 00:00:00 PDT

Yurts for Digital Nomads

Anil Madhavapeddy — Thu, 29 Apr 2010 00:00:00 PDT

The App Engine data collector for Personal Containers is coming on nicely, and is on track for an alpha preview release fairly soon. Working with AppEngine has been interesting; it’s got excellent availability and you can’t beat the price (free), but coding robust Python that doesn’t trip over the tight resource limits for individual requests, asynchronous tasks and queries is tricky. While it is good for small records such as my iPhone or Find My iPhone GPS traces traces, it doesn’t work so well with my gigabytes of photographs or decades of e-mail.

This confirmed our earlier intuition that there is no one perfect solution for personal data handling; instead, we need to embrace diversity and construct an infrastructure that can cope with change over the coming decades. Mobile programming has changed beyond recognition in just a few years, and cloud providers are specialising in different ways (e.g. PiCloud for simple compute, or EC2 for fancy services like elastic load balancing).

So to recognise this, we are building components that all interoperate with your personal data, keep it secure, and ensure it persists for more than a few years. Malte Schwarzkopf came up with the term “digital yurts”, and it’s stuck. We’ve written a draft paper about it, and would love to hear your comments and feedback on the approach.

There are some interesting recent trends that make doing this particularly important:

The New York Times wrote about the data-driven life increasingly influencing our decision making. Current sensor data such as GPS traces are just harbringers for the privacy disaster that would be information such as heart rates or your consumption habits getting into the public domain. (link via Derek Murray).
Facebook has announced a brand new API platform to get access to your information. The EFF has a fantastic timeline of Facebook’s Eroding Privacy over the last five years, to demonstrate how unsafe it is to trust your data to any third-party. We’ve started developing an information dump plugin for Facebook, but the API just changed mid-way and so it has to be started again (volunteers welcome!).
In the UK, the Digital Economy Act is an extremely controversial act that makes anonymity and privacy all the more important. We’re assembling an open-source dust cloud that integrates Tor into personal containers to automatically grant you anonymity as you communicate with your friends.

If you’re interested, join our group or contact me directly. At this stage, you need desire and the ability to hack code, but things are settling down over the next few months…

Pulling together a user interface

Anil Madhavapeddy — Thu, 15 Apr 2010 00:00:00 PDT

We’ve been hacking away on fleshing out the App Engine node for personal containers. We’re building this node first because, crucially, deploying an App Engine VM is free to anyone with a Google account. The service itself is limited since you can only respond to HTTP or XMPP requests and do HTTP fetches, and so its primary use is as an always-on data collection service with a webmail-style UI written using extjs.

Personal containers gather data from a wide variety of sources, and normalise them into a format which understands people (address book entries, with a set of services such as e-mail, phone, IM and online IDs), places (GPS, WOEID), media (photos, movies) and messages (Tweets, emails, Facebook messages). I’ll post more about the data model behind personal containers in a follow-up as the format settles.

The App Engine node has a number of plugins to gather data and aggregate them into a single view (see screenshot). Plugins include:

iPhoto extracts location (via EXIF), people present (associated via faces), and of course, the actual photograph.
Adium logs all IMs into a threaded chat view.
iPhone uses the backup files on a Mac to extract SMS messages, phone call records (and it could also get photographs and browsing history, although it currently doesn’t). An AppEngine tracker can also use FindMyIPhone to poll your iPhone regularly to keep track of your location without publishing it to Google or Yahoo (and hopefully in iPhone 4.0, we can operate as a background service at last!).
Twitter runs directly on AppEngine (authenticated via OAuth) and synchronizes with a Twitter feed.
SyncServices hooks into the MacOS X sync framework and initially subscribes to Address Book updates. This seems to be the first open-source sync alternative to the expensive Mobile Me, as far as I can tell. I’m planning to expand this to also subscribe to the full set of sync information (e.g. calendars).

I’m switching tacks briefly; we received an Amazon Research Grant recently and I’m building a node that runs as a Linux server to act as a longer-term archival and search server. This is being written in OCaml and uses Tokyo Cabinet (with Jake Donham’s excellent bindings) and so should be speedy and a useful alternative implementation of the HTTP REST interface. The plan is to automatically synchronize meta-data across all the nodes of a personal container, but store large and historical data away from expensive cloud storage such as App Engine.

There are lots more plugins in development, such as Foursquare and Gowalla OAuth collectors, an Android mobile application to upload location and contacts information, and Google GData synchronization. If you’re interested in one of these or something else, please do get in touch or just fork the project and start hacking!

A Personal Containers Marketplace

Richard Mortier — Fri, 09 Apr 2010 00:00:00 PDT

Personal Containers securely hold all your personal data to help maintain your privacy and security, and the current prototype enables that. However, to liberate the value contained within your personal data means allowing third-parties access to it somehow, so that they can provide services, advertising and other value-add. We think this is a pretty interesting problem, so we’ve added an initial design note with some thoughts on how we might solve it. Feel free to pitch in!

Opening a website

Anil Madhavapeddy — Mon, 29 Mar 2010 00:00:00 PDT

We’ve been working away at building a new type of database to help individuals keep reigns on their ever-increasing personal digital information. The first prototypes run freely on Google App Engine to gather your data behind-the-scenes, and we are working on more advanced versions that run on embedded devices and the cloud.

If you’re interested in keeping track of your personal data, you can start off with the installation instructions to clone your own version. After that, read up on the design of the system (which is still changing as we research new ideas around it). When you find something you want to fix, or add a new plugin data source, just clone the code and send us back fixes!