During the past year and a half, Palantir has been developing Palantir Video, a video analytics application that will allow Palantir Gotham users to better integrate video information into the rest of their analyses. The initial Palantir Video functionality will be limited to basic playback capabilities and the ability for a user to tag individuals and events within video data and associate those tags with information from other data sources in Palantir Gotham. As we continue to develop this capability, we will explore more advanced analytic capabilities, many of which may raise concerns about the potential effects of this kind of analysis on the protection of privacy and civil liberties.

Recognizing this, we convened some of our privacy and civil liberties advisors— a group of academics and advocates who we consult on new product developments that was recently formalized as the Palantir Council of Advisors on Privacy and Civil Liberties—to discuss video analytics. Over the course of several blog posts, we will present the issues raised by the group concerning video analytics in general and Palantir Video in particular, as well as how we might be able to use Palantir’s capabilities to support policies governing the use of video data that better protect privacy and civil liberties.[1]

In this post we will explore—at a very basic level—the global growth of video surveillance use, the current law on the use of video surveillance, and the privacy and civil liberties concerns that can accompany the deployment of video surveillance capabilities.

The Growth of Video Surveillance

The use of video surveillance has grown substantially in the last decade. In the United States, more and more major metropolitan areas have created CCTV systems. Authorities typically create these systems by both installing new cameras and tapping into existing surveillance networks operated by transit authorities, schools, and even private companies.[2] In the United Kingdom, which began using CCTV in the 1990s, a 2011 report estimated that there are more than 1.85 million cameras operating in the country—one for every 32 U.K. citizens. This includes not just law enforcement run cameras but also private sector systems.[3] A recent market research report predicted that the global CCTV market will reach around $23.5 billion by the end of 2014, which, the report suggests, represents a compounded annual growth rate of 20.5%.[4]

The rapid growth of CCTV systems is fueled by their perceived utility in supporting effective law enforcement efforts. Law enforcement officials argue that the use of video surveillance can deter crime, improve officer response time, and facilitate the after-the-fact investigation of crimes.[5] They frequently cite anecdotal evidence of particular crimes that have been prevented or solved with the help of CCTV.[6] Studies of CCTV effectiveness are sometimes less enthusiastic, generally finding “a modest but significant desirable effect on crime”[7] or at best suggesting that the impact of CCTV has been “variable.”[8]

The Current State of Law Governing Video Surveillance

When trained on public spaces, CCTV surveillance systems are minimally constrained in their use. Indeed, one privacy expert concludes, “Meaningful legal strictures on government use of public surveillance cameras in Great Britain, Canada, and the United States are nonexistent.”[9] In the United States, the question of the legality of public surveillance was effectively settled in United States vs. Knotts, in which the Supreme Court considered the legality of tracking a car’s movements with an electronic beeper. In its Opinion, the Court concluded that “a person traveling… on public thoroughfares has no reasonable expectation of privacy in his movements from one place to another.”[10] The private sector use of surveillance cameras in public spaces is similarly unconstrained, with the only real restriction coming from so-called “peeping Tom” statutes that criminalize filming or recording of certain activities where a person has some expectation of privacy.[11] The International Association of Chiefs of Police has developed guidelines for the use of CCTV for public safety and community policing, and some jurisdictions have adopted these or similar guidelines.[12] While useful, these guidelines are not legally binding.

The recent Supreme Court decision in United States v. Jones may indicate a potential avenue for judicial constraint of ubiquitous CCTV monitoring. Some commentators have suggested that a reading of the various opinions in that case suggests a majority of support for the “mosaic theory” of privacy. The theory argues that while the surveillance of individual acts taking place in public may not be constitutionally protected, “the collective sum of the different acts over time amount to a search” and therefore such surveillance should be subject to the Fourth Amendment.[13] As CCTV becomes more and more prevalent, one could envision a scenario in which the use of cameras to track an individual over an extended period of time is analyzed under the mosaic theory. However, any judicial adoption of this theory will have to await the next relevant case, which could be years in coming.

Video Surveillance and Threats to Privacy and Civil Liberties

The use of video surveillance systems by government agencies in a public space raises a number of privacy and civil liberties concerns such as:

• Generally speaking, surveillance can have a negative societal effect by inducing individuals to conform to societal norms and pushing individual choices towards the “bland and the mainstream.”[14]
• Even when in a public space, there is a “state of privacy” that flows from a degree of anonymity.[15] In other words, people behave differently when they are among strangers than they might when they are with people who recognize them. CCTV monitoring, combined with video analytics that facilitate identification, could diminish an individual’s sense of anonymity thereby inhibiting his or her freedom of action in a public space.
• CCTV surveillance is generally indiscriminate. It is “often akin to a dragnet search, which can ensnare a significant amount of data beyond that which was originally sought.”[16]
• Given this indiscriminate nature, there will be a temptation for substantial mission creep in which cameras deployed for one reason (e.g., to prevent and solve crime) become tasked with other missions that may be substantially different from this original intent. Such function creep is often not subject to the same public scrutiny as the initial camera installation. This is not a new phenomenon in surveillance. As observed by the Church Committee four decades ago, “We have seen a consistent pattern in which programs initiated with limited goals, such as preventing criminal violence or identifying foreign spies, were expanded to what witnesses characterized as ‘vacuum cleaners,’ sweeping in information about lawful activities of American citizens.”[17]
• As with any surveillance tool, CCTV can be abused by bad actors within an organization. Individuals with access to CCTV data may use the tools at their disposal to steal information, stalk an ex-spouse, or otherwise abuse their power. Any surveillance system must be designed with an awareness of this risk, however uncommon these abuses may be in actual practice.

With these issues in mind, we turned to our advisors for help. How do video analytics contribute to these P/CL concerns? Are there other concerns raised by the development and deployment of sophisticated data analytics? How could we design Palantir Video to help mitigate these concerns?

We learned a great deal from our discussion and will share some of that with you in our next few posts.

***

[1] Not all members of the Palantir Council of Advisors on Privacy and Civil Liberties attended this discussion. Participants are free to discuss the meeting publicly if they choose, provided that they do not disclose any proprietary or otherwise confidential information (no information was so designated in this discussion). In order to encourage an open and frank exchange, Palantir will not publicly attribute any comments to individuals or organizations participating in the meetings. ↩

[2] Alex Johnson, “Smile! More and more, you’re on camera,” MSNBC, June 25, 2008. http://www.msnbc.msn.com/id/25355673#.UH285Ib6p8F. ↩

[3] “Big Brother is DEFINITELY watching you,” Daily Mail, March 3, 2011. http://www.dailymail.co.uk/news/article-1362493/One-CCTV-camera-32-people-Big-Brother-Britain.html. Others estimate the number of cameras to be as high as 4.2 million. See, John Woodhouse, CCTV and its effectiveness in tackling crime, House of Commons Library, July 1, 2010. http://www.parliament.uk/briefing-papers/SN05624.pdf. ↩

[4] Nicholas Bombourg, “Global CCTV Market Forecast to 2014,” PRNewswire, March 13, 2012. http://www.prnewswire.com/news-releases/global-cctv-market-forecast-to-2014-142456805.html. ↩

[5] Department of Homeland Security, CCTV Developing Privacy Best Practices, December 17/18, 2007. http://www.dhs.gov/xlibrary/assets/privacy/privacy_rpt_cctv_2007.pdf. ↩

[6] See e.g., Allison Klein, “Police Go Live Monitoring D.C. Crime Cameras,” Washington Post, February 11, 2008. http://www.washingtonpost.com/wp-dyn/content/article/2008/02/10/AR2008021002726_pf.html.↩

[7] Welsh, Brandon C. and David P. Farrington, Effects of Closed Circuit Television Surveillance on Crime, Campbell SystematicReviews, 2008.↩

[8] Martin Gill and Angela Spriggs, Assessing the impact of CCTV, Home Office Report, February 2005. ↩

[9] Christopher Slobogin, Privacy at Risk (Chicago: University of Chicago Press, 2007) 89. ↩

[10] United States v. Knotts, 460 U.S. 276, 281 (1983).↩

[11] Slobogin 89.↩

[12] https://www.theiacp.org/LinkClick.aspx?fileticket=L8L%2BrgmpfWQ%3D&tabid=423. See also, NYPD Domain Awareness System Public Security Privacy Guidelines (http://publicintelligence.net/nypd-domain-awareness-system-public-security-privacy-guidelines/), UK CCTV Code of Practice (http://www.ico.gov.uk/~/media/documents/library/data_protection/detailed_specialist_guides/ico_cctvfinal_2301.pdf).↩

[13] Orin Kerr, “What’s the Status of the Mosaic Theory After Jones?” The Volokh Conspiracy, January 23, 2012. http://www.volokh.com/2012/01/23/whats-the-status-of-the-mosaic-theory-after-jones/.↩

[14] Julie Cohen, Examined Lives: Informational Privacy and the Subject as Object, 52 Stanford L. Rev. 1373, 1426 (2000).↩

[15] Allan Westin, Privacy and Freedom, (New York: Atheneum, 1967) 31.↩

[16] Dan Solove, Nothing to Hide, (New Haven: Yale UP, 2011) 179.↩

[17] Select Committee to Study Governmental Operations with respect to Intelligence Activities, “Intelligence Activities and the Rights of Americans,” U.S. Senate, April 26, 1976, pp 3 – 4.↩

One of the most difficult challenges for cyber security analysts is navigating through vast quantities of network data, which can approach petabyte scales and is often distributed across many disconnected systems. In this demonstration, we show how an analyst can use the Palantir Cyber solution to detect beaconing, a network behavior suggestive of malware, by querying multiple databases at a large institution in a matter of seconds. As fraudulent patterns are uncovered, analysts can automate these searches into regularly run jobs, serving as proactive alerts of malicious activity that are fed into our new prioritized inbox interface, powered by Hadoop. Finally, these alerts can be shared between analysts through Palantir Gotham’s collaboration application, which enables the rapid exchange of information within and across institutions to diminish cyber security threats.*

*While this demonstration is based on a typical investigation workflow, the data is simulated and names were randomly generated. Any resemblance to real people or entities is coincidental.

Congratulations to the winners of this year’s Palantir Scholarship for Women in Technology! This year, scholarship applicants wrote essays addressing why they were pursuing a technical degree, what can be done to increase the number of women in technical fields, and what ways technology can be used to promote opportunities for women worldwide. The finalists were flown to Palo Alto to visit Palantir’s HQ. Morning interviews and a product demo paved the way for a lunch with our Philanthropy Engineers, a tour of Palo Alto, and roundtable discussions with Directors Bob McGrew and Michael Lopp. To cap off the day, the women joined fellow girl geeks at a Bay Area Girl Geek Dinner, the second to be hosted by Palantir.

Winners ($10,000 Scholarship)

Carrie Cai – MIT, MS/PhD Candidate

Carrie is currently working on her MS and PhD in Computer Science at MIT, where she is focusing on user interface design and spoken language systems. After graduating from Stanford with a BA in Human Biology and a MA in Education, she returned to school to pursue Computer Science.

Excerpt from Carrie’s Essay Submission:
“In my view, computer science has a tremendous potential to improve the lives of those facing barriers to education and language acquisition. My dream is to create software that will enable others to more easily communicate and comprehend, through developing language assistive technologies and digital interfaces for learning.”


Anna Kornfeld Simpson – Princeton, 2014
Anna is pursuing a B.S.E in Computer Science at Princeton. She is interested in information security, operating systems, networks, robotics, and getting more women excited about engineering and computer science.  Anna co-founded Princeton’s Women in Science Colloquium in 2010 to connect female students and faculty in math, science and engineering.

Excerpt from Anna’s Essay Submission:
“…when I think about the strength of the truly diverse communities I have been in, I know that those with closed minds are the minority. If I am the only woman in the room, then my friend who joins next year will not be. By reaching out to other women interested in computer science, engineering, and robotics, we can build a community that supports and inspires everyone.”

Runners Up ($7,000 Scholarship)

• Nicola Dell – University of Washington, PhD Candidate
• Aakanksha Sarda – MIT, 2014

Finalists ($2,000 Scholarship)

• Elena Frey – Stanford, 2015
• Kathryne Hawthorne – University of North Carolina, 2013
• Katie Kuksenok — University of Washington, PhD Candidate
• 
Omosola Odetunde – Stanford, 2013
• 
Pratiksha Thaker – MIT, 2014

Semi-Finalists

• Willa Chen — Princeton, 2013
• Elaine Chen — Stanford, 2013
• Bonnie Eisenman — Princeton, 2014
• Darya Filippova — Carnegie Mellon, PhD candidate
• Kirsten Koa — University of California, San Diego, 2014
• Casatrina Lee  — Stanford, 2014
• Jennifer Sleeman — University of Maryland, PhD Candidate
• Kyle Rector — University of Washington, PhD candidate
• Caitlyn Seim — Georgia Tech, 2013
• Amy Tai — Harvard, 2013

For information about next year’s scholarship, check out www.palantir.com/scholarship in the Summer of 2013!

When looking for a job after graduation, many computer science or engineering majors assume that software development is their only career path. At least that’s how I felt at first. After all, my curriculum was dominated by courses on one or another aspect of software development. But when considering my career options, I realized I found coding interesting because I could use it to manipulate my computer, not because I found coding fascinating in and of itself. For example, I’d rather write code to organize the files in my music collection than write code to create a new filesystem. I routinely wrote tools to fetch and compile some cool new open-source project, but I was never especially interested in adding more features to those projects. When my network router broke at home, I enjoyed learning tools like ping, netstat, and tcpdump, but I didn’t really want to extend the core functionality of those utilities.

Those things that really interest me tend to be smaller in scope than full development projects. Once I’ve explored a tool or piece of code to solve a problem, I’m pretty much ready to move on to the next challenge. I’m like an explorer who is always searching for something new to learn and to satisfy my innate curiosity.

Server-Side Quality Engineers (SSQEs) at Palantir are explorers for these same reasons. We’re interested in exploring things like distributed systems, Linux servers, and databases. We want to learn how they work so that we can manipulate things around them. For example,

• Knowing how Palantir Gotham deals with data-scale (sharding) and user-scale (mirroring) allows us to configure clusters that resemble customer deployments.
• Knowing how Linux behaves during entropy starvation (and being able to find the cause) allows us to efficiently use our server hardware.
• Knowing how Oracle handles various SQL statements allow us to spot slow database performance.

If my experience resonates with you at all, read on to learn about the role of SSQEs during various stages of the Palantir software development cycle.

Feature Vetting and System Architecture Review

Before each iteration of the Palantir software development cycle, we spend a week with our Software Engineer counterparts to understand what they are building. We want to know what features they will be implementing, how they plan to implement those features, and which customers those features are targeting. Developers should know the answers to questions like “how many users does this feature support simultaneously?”, “does it interact with any existing features?”, “what if the system crashes while this operation is still running?”, “is this operation idempotent if someone runs it twice?”, and so on. We will also talk to our Forward Deployed Engineer counterparts (also known as BD or Business Development) to make sure the use cases we envisioned match real customer use cases. This is also a good time to discuss customer hardware configurations, special data characteristics, and other deployment-unique cases.

When necessary, we ask how a particular piece of system architecture, protocol, or algorithm works to ensure that we understand how to test and manipulate it. We also try to discover non-obvious corner cases early in the cycle. Since the best idea always wins at Palantir, planning week is the ideal time to ask any (and a lot of) questions about a feature or a system. Making changes at this point is MUCH cheaper than later in the cycle.

New Feature Testing

Explorers of old filled their notebooks with detailed drawings of mountains, rivers, plants, and animals. Like those explorers, we write down our new-feature discoveries into our version of those notebooks (test plans) so that others can follow in our testing footsteps. Test plans initially contain information about how a feature works, for whom it was built, and other detailed notes from planning week.

As new feature development progresses, developers hand off their code in discrete milestones. We test each milestone and further refine the test plan. For example, in the test plan, Quality Engineers describe how to set up a feature, propose reasonable data sets, write testing instructions, and record expected results. Developer milestone hand-offs and testing of the milestones continue in this passing-of-the-ball fashion until the end of the new feature period.

System Debugging

During new feature testing, we expect the Palantir Gotham application to be unstable as many developers check in code simultaneously. One of our duties as Quality Engineers is to report any unexpected behaviors as bug reports so that developers can fix them. Quality Engineers at some other companies call their job done at this point. Palantir Quality Engineers take the extra steps to diagnose whether an issue comes from the system (Linux settings, Oracle configurations, network issues, CPU, RAM, IO contention, etc.) or from the product itself. This extra bit of effort increases quality in the bug report, and as the bugs are resolved, in the product.

But how do SSQEs know what to do to debug these things? Well, most SSQEs needed to do this type of debugging in the early days of Linux in order to even use the operating system, so they’ve gained tons of experience doing it. Since hardly anyone else was using Linux back then, the early pioneers frequently found problems that no one else has ever come across. Today, using Linux is easier than ever before. On the negative side, unless you’re naturally really curious, you probably haven’t been exposed to this type of debugging because using Linux is so easy. Explorer tip: If you want to bulk up your Linux knowledge to give yourself an edge during the interview process, read ‘How Linux Works: What Every Superuser Should Know’ by Brian Ward.

Exploring the Server Debug Challenge

If the above sounds interesting or even exciting to you, let’s talk about how you can join the SSQE explorers. We begin the process by giving you an opportunity to explore a real system. This is the Server Debug Challenge. While this challenge is optional, it presents a realistic system debugging scenario that gives you a chance to show off your exploration skills. It’s also a way for you to set yourself apart from the many applications that we receive. If your report details the steps you took to correctly find the problem, we definitely want to talk to you. Whatever the outcome, we’re pretty sure you’ll have fun playing with it.

Girl Geeks from around the Bay rang in the new year and broke in Palantir’s new event space with Girl Geek Dinner #32, which focused on philanthropy, technology, and the opportunities presented by both for women worldwide.

Lucy Bernholz, visiting scholar at Stanford University’s Center on Philanthropy and Civil Society, opened the night with her thoughts on the future of philanthropy. She argued that as the quantity and quality of collected data increases, philanthropy will move from being an industry fueled by people’s passion to an industry driven by data. She urged the Girl Geeks in attendance to contribute to this shift by finding ways to share and engage data to drive change in the social sector.

Bernholz’s vision of the future is Chelsea Geyer’s reality. Geyer is a Project Developer at Resolve, which uses software donated by Palantir to understand and analyze the Lord’s Resistance Army (LRA), the fundamentalist religious group responsible for brutal human rights violations across large parts Africa, and made infamous by the Kony 2012 video. Resolve uses data about past LRA attacks to better understand their movements and try to project who, where, and how the LRA will attack next. By analyzing the data with Palantir’s software, Geyer and Resolve are creating an ever-evolving trail of evidence that could be used if members of the LRA were ever brought to the International Criminal Court.

While there was heavy emphasis on how technology could be used for good throughout the night, Mary Quigley, a Palantir Gotham user, touched on how technology can increasingly be used for bad. At the major financial institution where she works, Quigley is responsible for preventing cyber attacks, including denial-of-service attacks, which would cripple the institution’s network if successful. Her experiences serve as a reminder to our community to continue working to improve the tools that can be used to counter bad actors and their constantly adapting tactics.

To end the night, Maura O’Neill, Chief Innovation Officer at USAID, told the Girl Geeks about how much technology has evolved in her lifetime and what that has meant for women worldwide. She also spoke about a problem that technology has so far not been able to solve: equal treatment in the workplace. O’Neill shared her own experiences with this problem. In the past she was overlooked for promotions that her male colleagues received. Just last month, she was brazenly asked to take notes at a dinner reception. O’Neill urged the women in the room to pave a path that their mothers would be proud of, and that their daughters could continue to march along.

It was a powerful end to an evening of compelling talks. The night continued with demonstrations of Palantir’s software, flip-book making, and conversations over cocktails.

Bay Area Girl Geek Dinners, hosted by companies around the Bay, present chances for women in technology to meet, socialize, and learn from each other’s experiences. Palantir hosted its first Girl Geek Dinner in May 2011, which featured Linda Krieg, Vice President and Chief Operating Officer of the National Center for Missing and Exploited Children. For information about the next Girl Geek Dinner, visit: http://www.bayareagirlgeekdinners.com

With development times of ten years or more and costs of over $1 billion per new medicine, pharmaceutical R&D is an expensive, lengthy, and risky process. Enterprises can minimize the risk associated with drug development programs by better understanding the universe of data relevant to particular diseases, targets, and drug candidates.

Unfortunately, achieving this level of understanding is technically challenging, as the data is typically scattered across public and private databases and stored in a wide range of formats. Furthermore, novel research data is noisy and can be fraught with contradictions. Serious data security and access control concerns only add to this complexity.

The Palantir Gotham data fusion platform helps pharmaceutical companies overcome these R&D challenges by integrating public and private data of nearly any type and empowering secure collaboration both within and beyond the enterprise.

In this three-part series of videos, you will see how the Palantir Gotham platform facilitates rapid, intuitive, and comprehensive exploration across a variety of data sets. We focus on just a handful of ways in which Palantir removes the friction between pharmaceutical researchers and their data, allowing them to (1) discover connections between assays performed by different teams, (2) evaluate evidence for a drug-protein interaction, and (3) capture the investigative process for future use.

For these demonstrations we have integrated data from public sources including ChEMBL, PubMed, KEGG, PDB, and PubChem. From high-level literature reviews to the analysis of specific variations in a genetic sequence, Palantir Gotham increases efficiency in the drug development process and amplifies signal strength for key pipeline decisions.

Part 1: To start, we use Palantir Gotham to collaboratively model biological relationships and assess their certainty in order to better understand the risk associated with targeting a particular pathway.

Part 2: Next, we explore new experimental results that were produced by an external team. We investigate their underlying scientific rationale and apply these insights to my own drug development research.

Part 3: Finally, we investigate structured assay data and quickly drill down on a compound of interest. We use a variety of techniques to make sense of complex experimental results in both structured and unstructured formats.

Drug development is just one of many potential pharmaceutical applications of the Palantir Gotham platform. Our goal is to solve your organization’s most challenging data problems across the pharmaceutical pipeline. To learn more, read about our Pharma solution and contact our health team: helix@palantir.com.

Like the other domains in which our software is deployed, investigative journalism relies on the insight and judgment of human experts. Armed with the right technology and enough data, these experts can achieve extraordinary outcomes. On the Philanthropy Engineering team, we’ve been fortunate to collaborate with passionate, truth-seeking journalists on several inspiring projects, many of which are still making headlines.

In 2011, The Center for Public Integrity (CPI) published The Truth Left Behind, a report on the kidnapping and murder of Daniel Pearl. The report was the product of a three-year investigation known as The Pearl Project, which was carried out by students and faculty in Georgetown University’s Journalism program in conjunction with CPI. We donated our software to help Pearl Project investigators build out networks of people and organizations suspected to be involved in the murder. In the CNN video below from 2011, lead investigator Asra Nomani described some of the report’s findings. You can watch videos of some of the analysis on our Analysis Blog.

Pakistani authorities have recently arrested Qari Abdul Hayyee, a former leader of Lashkar-e-Jhangvi, in connection with the case. The Wall Street Journal notes that The Pearl Project’s report named “a Lashkar-e-Jhangvi operative known as Adbul Hayee–a different spelling of the name believed to be the same person–as one of the men who guarded Mr. Pearl after his kidnapping.” You can read more about our involvement with The Pearl Project here.

The Pearl Project is just one of several investigative projects to which we have donated our software and analytic support. In 2009, we helped CPI to analyze almost five million home mortgage loans that originated between 2005 and 2007 as part of an investigation into subprime lending practices. CPI was able to show that a small number of lenders were responsible for a disproportionately high number of the riskiest loans. You can read more of their findings in “Who’s Behind the Financial Meltdown?” and you can learn more about the methodology behind the analysis at CPI’s website.

Last year, we worked with CPI’s International Consortium of Investigative Journalists to support their investigation of illicit human tissue trafficking networks. ICIJ broke the story–called “Skin and Bone”–at the Google Ideas Summit on Illicit Networks in July 2012, and in August it won a Sidney Award from the Sidney Hillman Foundation. Their reporting prompted Congress and the Defense Department to begin their own investigations into their tissue supply networks. You can learn more about the methodology behind the analysis at ICIJ’s website.

Most recently, we collaborated with CPI on “Cracking the Codes,” an analysis of Medicare claims data that uncovered more than $11 Billion in Medicare billing errors and possible abuses by US doctors nationwide. This project won first place in the 2012 International Reporters and Editors (IRE) Philip Meyer Awards. You can learn more about the methodology behind the analysis at CPI’s website.

We’re incredibly proud to have been a part of this award-winning work, and to have helped investigative journalists tell stories that have had a real, definite, and positive impact on the world. As the data-driven journalism and computer-assisted reporting movements continue to grow and evolve, we look forward to seeing more cooperation between software engineers and investigative journalists.

It is estimated that there are 21 million people enslaved worldwide.[1] Human trafficking is the third largest global illicit trade and is growing at a higher rate than weapons and narcotics. Men, women, and children are victims of trafficking internationally as well as within the United States.

Polaris Project, Liberty Asia, and La Strada International are recipients of a $3 million dollar Global Impact Award given by Google to support nonprofit organizations that use technology to initiate disruptive solutions in their sector. Together, these organizations are launching the Global Human Trafficking Hotline Network, an initiative seeded by Google Ideas to share data and improve coordination between local hotline efforts. Palantir is pleased to announce our collaboration with Polaris Project, improving the network’s ability to manage and analyze the large amounts of disparate data culled from its human trafficking hotline.

Polaris’ hotline has received 72,000 trafficking-related calls, reported more than 3,000 trafficking cases to law enforcement officials and assisted nearly 8,300 victims of modern-day slavery. The donation of our analytical software platform–with built-in privacy and civil liberties safeguards–as well as our ongoing training and support augments Polaris’ existing efforts to disrupt illicit trafficking networks and provide much-needed assistance to victims.

The video below, produced by Google Ideas, describes the Global Human Trafficking Hotline Network in greater detail.

Polaris Project is just one of the many ways in which Palantir’s Philanthropy Engineering efforts have assisted the fight against illicit networks, human trafficking, and child exploitation.

***

[1] International Labor Organization 2012 Global Estimate of Forced Labor.

The Privacy and Civil Liberties Team was in Brussels, Belgium, in January for the Computers, Privacy, and Data Protection conference, a three-day meeting of academics, NGOs, government officials, and corporations concerned about legal and technological issues related to the protection of privacy. In addition to a number of interesting panel discussions, the conference was an occasion for conversation. It provided a wonderful opportunity to get a global perspective on privacy and data protection issues from a wide variety of viewpoints. In the course of swapping theories and cases studies with other privacy and technology wonks, I got an opportunity to introduce many people to Palantir for the first time.

But the conference was also an occasion to appreciate the limits of conversation. As I delivered my now well-honed explanation of what Palantir does, and why and how we build technology that can enable better protection of privacy and civil liberties, my interlocutors typically reacted in one of two ways. Some were greatly appreciative of our recognition of and approach to these complex issues, while others were highly skeptical of the authenticity of our commitment and our ability to actually deliver on our promises. Interestingly, both camps often ask some version of the following question: “You have built a very powerful analytic platform and you work with many customers who could do serious damage if they misused it. Why should we trust you when you say you are actively working to protect privacy and civil liberties?”

My response: “You shouldn’t just trust what we say. And we are not asking you to.”

It’s the same response I give to recruits and fellow Palantirians who are similarly committed to protecting PCL. If Palantir were being insincere about its commitment to privacy and civil liberties, we would risk alienating the very people who are most essential to helping us succeed in our mission. Everyone here wants to be proud of what they build, which means that they do not want to see their work misused to violate fundamental liberties. Palantirians, no less than the privacy community, want reassurances about the kind of work we do and the seriousness of our commitment to doing the right thing. A simple profession of our PCL stance isn’t enough. Talk is cheap.

That’s why we continue to take action:

• We’ve invested in a (growing!) Privacy and Civil Liberties Team, which is dedicated to working with customers to help them implement law and policy designed to protect privacy and civil liberties, and to encourage a higher ethical standard in the use of our product.
• We have formed the Palantir Council of Advisors on Privacy and Civil Liberties to help us navigate tricky legal and ethical questions and advise us on the future development of the product so that we can address potential privacy and civil liberties issues by “baking in” protective capabilities.
• We provide financial support to privacy and civil liberties advocacy organizations, and we are proud to sponsor events such as the Amsterdam Privacy Conference and the Privacy Law Scholars Conference.
• Additionally, Professor Dan Solove, an internationally recognized privacy scholar, has helped us to design the privacy and civil liberties training program that all new Palantir hires will be attending.

We also try to be as transparent as possible in describing the capabilities of our software. Our Analysis Blog demonstrates the myriad ways in which our technology is being used at deployments around the world. We proactively work to educate policymakers, academics, and advocacy groups about these capabilities and in order to identify the questions that should be asked of us and our customers. Are access controls being used at a granular level to protect information? Are audit logs regularly reviewed for indications of potential misuse? Could a federated system be built instead of centralizing data in a single database? We also seek input on how these capabilities could be improved to better address potential privacy and civil liberties concerns.

These are the actions we have taken and will continue to take to demonstrate the depth of our commitment to protecting privacy and civil liberties—to earn the trust of those who are similarly committed, not only within the PCL community but within our own company.

I can testify to the fact that Palantirians reach out to the Privacy and Civil Liberties Team on a daily basis to seek advice, comment on our work, offer to help, or simply ask us to explain particular issues in the privacy and civil liberties world. Every Palantirian is trained to look out for “red flags” at deployments that might indicate activities that are antithetical to our commitment to privacy and civil liberties. We all have a responsibility to “watch the watchers” (and those who enable the watchers), ask the questions that need to be asked, and demand changes when we see something we don’t like. So, like our fellow Palantirians, don’t just trust us. Look at what we do; don’t just listen to what we say. Question us, criticize us, and challenge us.

We work with organizations that analyze many different kinds of data from many different sources, each of which is typically governed by its own access control or security policies. Our customers need to be confident that their analysts are handling this sensitive data appropriately, and that their auditors can identify and take action against any data misuse.

We’ve developed two broad sets of capabilities to meet these needs:

1. Precision access control capabilities that support multi-level security policies. Palantir Gotham enables users with multiple and varying access permissions to interact appropriately with data with multiple and varying access restrictions. Users see only the data they are authorized to see, while operating in a shared, collaborative environment with other users who may have different access permissions.
2. Audit logging, data integration, and analytic capabilities that support the investigation and monitoring of data use. Palantir Gotham automatically generates an immutable audit log of all user activity, which can then be integrated back into the platform for analysis. By integrating additional data sources pertaining to data use at their organizations – everything from system log-ins to network activity, OS activity, and physical sensors – auditors can investigate potential instances of data misuse and actively monitor for suspicious activity as it occurs.

You can watch a demonstration that highlights access controls here. In this post we will focus on the second set of capabilities, and demonstrate how organizations can use our platform for effective audit log analysis.

Reactive investigations

With Palantir Gotham, an auditor can respond to potential data misuse or security incidents by discovering links between audit events (searches, object loads, data exports, etc.), user profiles, and administrator actions within the system. Even when dealing with low signal-to-noise data like audit logs, auditors using Palantir Gotham can find answers to their most important questions. Who has seen data X and when? What has user Y seen? Has user Y seen data X? Once this information has been surfaced, auditors can explore answers to subtler questions like: What was user Y’s intention in investigation X?

In this video*, the platform is used to trace the source of a sensitive information leak. An auditor can investigate the source by looking at audit events related to the leaked information (in this case, the “William Haynes” object). By using various search and analysis tools, the auditor can then rapidly focus his investigation on audit events that seem suspicious (e.g., those occurring outside normal business hours, or involving abnormal interactions with the leaked information). By cross-referencing these events with other information like badge swipe data, the auditor can discover which particular user profiles are implicated in the leak.

In this second video*, an auditor investigates data use by a particular user and finds evidence of unauthorized data access. He knows the user isn’t authorized to access information on US citizens, but finds evidence that the user has both accessed and edited data relating to a US citizen. He traces the cause of this unauthorized access to an administrator who made changes to the access controls governing the data.

Whether the investigation begins with a particular user or with a particular piece of data, Palantir Gotham supplies the audit logging, data integration, and analysis capabilities that auditors need to find the root causes of data misuse.

Proactive monitoring

Being able to track down the origins of past incidents of data misuse is crucial to maintaining user accountability and oversight, but it would be even better if auditors could stop data misuse as it happens, in real time. Palantir Gotham also provides a set of monitoring capabilities that enable auditors to rapidly detect and shut down suspicious data use. Search feeds and lead-generating algorithms turn massive-scale audit logs into actionable intelligence – information that can be used to mitigate potential threats quickly, before they can do great harm.

Ensuring appropriate data use

We are committed to building products that make our users better at the most important work they do. Palantir Gotham empowers administrators and auditors with tools to enforce compliance with applicable data protection policies. But these safeguards are only effective if they are put to good use, which is why we are also committed to working with our customers to help set up access controls and implement audit practices that can ensure appropriate data use at their organizations.

* While these video demonstrations are based on typical investigation workflows, the data is simulated and names were randomly generated. Any resemblance to real people or entities is coincidental.

On May 20, Palantir received four gold stars at the DI2E Plugfest in recognition of our platform’s demonstrated compliance with the Defense Intelligence Information Enterprise (DI2E) standards.

The DI2E Framework defines a set of standards that are meant to ensure interoperability between data systems across the Defense Intelligence Community. The DI2E SvcV–4 (the most recent DI2E “Services Functionality Description”) currently lists standards for 120 different services (.xls) across three different levels: Infrastructure Services, Common Services, and Mission Services. At Plugfest, Palantir was recognized as being compliant with over 90 of these service standards, spread across all three levels.

This certification represents an official validation of the work we’ve been doing since our founding. As a Silicon Valley technology company, we believe that open software is better software. An open architecture, open APIs, and an open data model have been central to the design of our platform from its inception. For years, this openness has enabled our software to interoperate with our customers’ existing systems across multiple industries, from defense to global finance.

As the DI2E Framework standards and specifications evolve in the future, our software will evolve with them to ensure that we can continue to provide our Defense and Intelligence customers with the capabilities that are revolutionizing the way they do their work. Beyond the DI2E Framework, Palantir is also working closely with the US IC and DNI IC CIO to ensure our technologies are fully compatible with the IC ITE ecosystem. Stay tuned for more on this.

As Shyam Sankar, our Director of Forward Deployed Engineering, explains in the video below, our platform provides users with not just an intuitive user interface, but an entire intelligence infrastructure that allows them to securely collaborate on their hardest problems across organizational and geographical boundaries.

Whatever we build next, we will continue to provide products that combine the efficiency, reliability, and value of a commercially developed platform with the extensibility, flexibility, and interoperability of an open system.

Our condolences and best wishes go out to all those suffering in the wake of the Oklahoma tornadoes.

After several tornadoes ripped through Oklahoma last week, killing 24 people and injuring hundreds more, Team Rubicon deployed its veteran and civilian volunteers to the disaster site. Several of our Philanthropy Engineers are currently in Moore, OK, where they are equipping the men and women of Team Rubicon with Palantir Mobile and helping them coordinate their relief efforts.  Time Magazine’s Joe Klein also spent Memorial Day weekend working with Team Rubicon, and he recently shared the story on his blog. Check it out to learn more about the amazing work that Team Rubicon is doing, and visit TR’s blog, too, where many of their volunteers are sharing their own stories.

Stay tuned for more updates on how we’re helping Team Rubicon to bring the victims some relief.

While teams of veteran relief volunteers and Palantir engineers deployed to Oklahoma to provide relief in the aftermath of one of the widest tornadoes ever recorded, the U.S. House of Representatives Homeland Security Committee asked experts to testify at a panel addressing Emergency Preparedness, Response and Communications. On June 4, Palantir’s Jason Payne joined Google’s Matthew Stepka and others at the Subcommittee hearing to discuss the importance of open data portals and information sharing in disaster scenarios.

“We encourage governmental organizations to adopt a Silicon Valley approach to data interoperability – put the data out publicly in a robust, standardized, well-documented interface and let other organizations come up with innovative ways to leverage the data,” Payne stated.

In January, Subcommittee Chairwoman Susan Brooks visited Silicon Valley to explore how new technologies are transforming emergency preparedness and response efforts. “On that trip we met with companies that were on the leading edge of new technology that are also contributing to the preparation, response and recovery from disasters,” she stated in her opening remarks. “Two of these amazing companies are here today: Google and Palantir.”

In his comments to the panel, Stepka noted that data sharing and relief efforts are hindered when agencies release information in non-machine readable formats. “Our goal is to make it easier to get people the actionable information they need when they need it most.” Acknowledging that while more needs to be done to make information easier to acquire, the panelists praised the data sharing efforts of the National Oceanic and Atmospheric Administration, the National Weather Service and the Census Bureau.

Payne’s testimony highlighted our dual commitment to open data and privacy protection. “We believe that public data such as locations of shelters and medical facilities, power status reports, and satellite imagery should be available to all organizations and citizens. We also believe that sensitive information, such as names, dates of birth, addresses, phone numbers, social media posts, financial information and certainly medical information should be shared with only those with need to know that information, even within an organization.”

Open data, privacy, and civil liberties are important to us and we are pleased and honored to be a part of the public conversation addressing these issues.

Read more about our commitment to openness and disaster response and read the full committee testimony here.

Last Fall, we announced the creation of the Palantir Council of Advisors on Privacy and Civil Liberties (PCAP), a body of experts in this field who have been helping us to understand and address the complex privacy and civil liberties issues that arise in the course of providing sophisticated data analytics to our many customers around the world.

Today, we are pleased to announce that Nancy Libin will be joining PCAP.  Prior to her current position as a Partner at Wilkinson Barker Knauer, LLP, Nancy was the Chief Privacy and Civil Liberties Officer of the U.S. Department of Justice for over three years. She also worked as Counsel to then-Senator Joseph Biden on the Senate Judiciary Committee and at the Center for Democracy and Technology. We are excited to have someone with Nancy’s expertise on PCL issues in the law enforcement and federal agency space joining our esteemed group of advisors.

In other news, Jeffrey Rosen was recently named president and CEO of the National Constitution Center, a nonprofit, nonpartisan institution devoted to constitutional education and to providing a forum for discussion and debate on important issues of the day.  Unfortunately, Jeff’s new position imposes constraints that will make it impossible for him to continue to participate in PCAP. We are extremely grateful for the time Jeff was able to give us and for the quality of his contribution, and we wish him well in his new endeavor.

Lauren Chaparro and I were honored to be among the speakers at Strata RX 2012, O’Reilly’s conference on the use of big data in health care/medical field. Our talk was called “Doing Big Data All By Yourself: Interactive Data Driven Decision Making by Non-Programmers“.

I gave the first half of the talk, delving into the stark realities of big data implementations. The second half of the talk featured Lauren (a Palantir Forward Deployed Engineer on the health applications team) giving a live demonstration of workflows in a system that we recently implemented on top of Palantir Gotham, one of our data fusion platforms.

There is a lot of excitement in the marketplace around the availability and capability of big data processing technologies. But most of these technologies are building blocks, not complete solutions. Before health care providers can realize the promise of big data, they need to overcome at least three major challenges associated with implementing these technologies:

• Systems Engineering. The integration work required to deploy this technology at scale is non-trivial.
• Data Science. Someone needs to design the statistical analysis that can derive actionable insights from the raw data.
• User Experience Design: Big data implementations require a user interface that makes it easy for subject matter experts to ask questions of the data.

In the health care space, the end users are clinicians and researchers. Many big data implementations, however, feature interfaces best suited to data scientists and programmers. As a result, this technology is mostly used for doing aggregation and static dashboarding. This is a good start, but it falls short of the goal of putting the power to learn from big data into the hands of clinicians.

The punchline here is that the scarce resource in the big data domain (regardless of vertical) is talent – the talent to (a) do the complex system engineering and data science necessary to derive insights from data and (b) build the last mile of familiar, expressive, and interactive interfaces needed to truly take advantage of all that the data has to offer.

The second half of the talk focused on work we did in association with Center of Public Integrity. We put together a Palantir Gotham instance that integrated anonymized data from Medicare and various other data sources to show the potential of a fully integrated, interactive system.

The datasets involved were:

• Medicare data representing 100 million claims, 1 billion medical procedures, 30 million individual beneficiaries, and 700,000 physicians.
• Data from the National Plan Provider Enumeration System, used to standardize identifiers across payers and providers.
• Data from the Dartmouth Atlas Project – a well-curated collection of hospital-specific performance data.
• Data from PubMed, representing 22 million biomedical journal articles.
• Data from the Department of Health Human Services Office of the Inspector General composed of entities excluded from participation as Medicare providers due to past fraudulent behavior.
• Data from the US Census showing demographic trends across the country

Since patient privacy concerns are paramount with this sort of data, Lauren Chaparro used simulated rather than real data to give a live demonstration of how different subject matter experts could perform a number of different workflows inside a single system:

• Policy makers can explore answers to high-level questions around the supply and demand of hospice care given the reality of an aging population.
• Investigators can detect patterns of fraud in billing for hospice care and find individual bad actors.
• Doctors can look into the optimal treatment options and providers for a given patient.

We believe answering questions like those addressed in this demo is key to curbing waste, fraud, and abuse in our nation’s healthcare system and improving healthcare delivery. Through the integration of a variety of datasets at massive scale, our software can empower insurers, policy-makers, and physicians to pursue these kinds of hypotheses and derive actionable insights today, without turning to data scientists.

Lawrence Lessig, Roy L. Furman Professor of Law at Harvard Law School and the founder of the Stanford Center for Internet and Society, has long been at the forefront of thinking about how law and technology interact.  His influential book, Code and Other Laws of Cyberspace, included his now famous assertion that “Code is law.” He contends that the architecture of technology itself, at least as much as traditional legal structures, dictates how privacy and civil liberties are protected for those who use it. As he states in a recent interview with Bill Moyers, “We’ve got to think about the technology as a protector of liberty too.” Check out the full interview to hear Lessig discuss this theme in greater depth. He mentions Palantir’s PCL-protective capabilities at 13:30.

Lessig’s recognition of the important role played by technology in protecting privacy and civil liberties has had a significant influence on our own approach to these issues. We continue to look for ways in which technology and law can support each other by building effective analytic capabilities that protect privacy and civil liberties at the same time.

Team Rubicon’s William Gauntner uses Palantir Mobile client to provide immediate data to relief agencies in tornado-ravaged Oklahoma.

Time’s Nation blog just published a piece describing the work of Team Rubicon in Moore, Oklahoma in the wake of the powerful tornado that created a swath of devastation on May 20, 2013. Entitled, “How Vets and Military Technology Are Helping at Home,” the piece focuses on the work that the volunteers of Team Rubicon, many of them veterans, perform in doing disaster recovery operations. We’re honored to aid Team Rubicon in their absolutely essential mission of helping communities put themselves back together when natural disasters strike.

On Time’s Nation blog, “How Vets and Military Technology Are Helping at Home” describes Team Rubicon‘s disaster response work in the wake of the tornado that devastated Moore, Oklahoma on May 20, 2013.

From the piece:

Team Rubicon is currently testing a mobile platform that could dramatically improve disaster response.

Gauntner and his team are assessing properties using an Android smartphone loaded with software from Palantir Technologies, a Silicon Valley data analysis startup. Palantir partnered with Team Rubicon last year, fielding devices for volunteers in New York City after Hurricane Sandy, again in Illinois after the flooding this spring, and now in Oklahoma following two devastating tornadoes.

Together, Palantir and Team Rubicon are testing and improving Palantir Mobile. Using an Android smartphone, Team Rubicon volunteers provide critical knowledge about damaged areas in real-time by submitting the information to a central database.

Delivering advanced information management capabilities to disaster response operations continues to be a primary focus of our Philanthropy Engineering team. Our deployment to Moore, Oklahoma was the fourth in our partnership with Team Rubicon, which began with supporting their recovery operations in the wake of Hurricane Sandy. Along with a mobile client for disaster reporting, we provide a desktop client for rich, interactive analysis that is used at the base of operations to coordinate with volunteers in the field and to manage the overall recovery effort. We’re honored to aid Team Rubicon in their mission to help communities put themselves back together after natural disasters strike.

In late May, we held GovConUK II at the Renaissance St. Pancras Hotel in London. The event highlighted our expanding global footprint and the technical advances we’ve made since our first conference in the UK two years ago.

Returning GovCon presenter, Sgt. Pete Jackson, demonstrated how Palantir Gotham’s capabilities are used at the Los Angeles Sheriff’s Department. CEO and co-founder of Team Rubicon, Jake Wood, told the story of an unlikely partnership between an organization of volunteer veterans and a bunch of nerds. Over the course of the day, Palantir engineers shared their perspectives on different data problems, and demonstrated how Palantir’s products can be deployed against these problems to achieve extraordinary outcomes at the organizations where we work. Attendees saw how Palantir Gotham can help organizations prevent, detect, and harden their defenses against cyber attacks, and how Palantir Metropolis can enable analysts and decision makers to detect insider threats within their organizations.

For those of you who are interested in the content of the conference, some of the presentations are now available on YouTube:

GovConUK II Presentations

Palantir 101
Arlo Guneratne Bryer, Embedded Analyst, Palantir

For those who are completely new to Palantir or could simply use a refresher, this talk starts from scratch and provides a broad overview of Palantir’s origins and mission. A live demonstration of Palantir Gotham helps to familiarise newcomers with the product’s intuitive user interface and revolutionary analytical functionality, while highlighting the major engineering innovations that make it all possible.

Technical Challenges at the Enterprise and the future of the Palantir Platform
Bob McGrew, Director of Engineering, Palantir

Palantir continues to push boundaries at the enterprise and with this come exciting new product features. This talk covers three major enterprise environments and the technological advancements that they motivated. In the commercial security enterprise space, we have tackled massive data scale and low signal within these data. In the armed forces enterprise, we have addressed the challenges around geographical separation of groups who must work together with Nexus Peering, which continues to enable secure data sharing in ways that were not possible before. Lastly, in the Intelligence Community enterprise, we are supporting unprecedented user scale by ensuring platform resilience and availability and also by building a web-based version of our application that is easily accessible and usable.

Prepare, Detect, Respond, and Harden: Palantir Cyber in Action
Melody Hildebrandt and Sean Hunter, Palantir

As the threat of cyber attacks grows, so too does the scale of data collected that could inform organizations that are most at risk. This presentation covers how Palantir Gotham helps institutions across different industries handle petabytes of data, protect their most sensitive information, detect both internal and external threats, respond quickly to attacks, and harden themselves against future cyber attacks.

A Story of Accidental, World-Changing Collaboration
Jacob Wood, CEO Team Rubicon

Palantir donated its software and the expertise of its Philanthropy Engineering team to Team Rubicon to provide relief to those affected by Hurricane Sandy last Winter. Hear how Team Rubicon, a non-profit relief organization that dispatches volunteers from its network of nearly 5,000 military veterans, is using Palantir Gotham and Palantir Mobile to revolutionize disaster relief by responding quickly and efficiently after natural disasters strike.

Humans, Data, and the Culture of Too Much Information
Michael Lopp, Palantir

“Big Data” is a phrase that is often used, but poorly defined. This talk explores the history and emergence of “Big Data,” how humans have responded to the rapid, exponential growth of data, and why nerds are uniquely capable of turning it into useful information.

One Step Ahead: Internal Threat Detection & Prevention Within Your Organization
Alexa McLain and Mike Reilly, Embedded Analysts, Palantir

Sometimes, the biggest threat and risk faced by an organisation comes from inside its own walls. Traditional IT solutions prove to be insufficient when analysts face an overwhelming amount of alerts and have inadequate tools to investigate and understand behaviours in context. This presentation explores the threat landscape and where Palantir Metropolis can help connect the dots.

Leveraging OSINT: Introducing Palantir Torch
Quentin Spencer-Harper and Ben Duffield, Palantir

Recent years have seen an explosion in the volume of social media data produced on the Internet. As the use of social media expands, in aggregate, it can provide access to critical information in real-time. In this presentation, we demonstrate Palantir Torch, which integrates billions of tweets at sub-minute latency to quickly discover trending conversations, understand their content, and visualise their evolution over time.

Palantir on the Beat: The Technology that Empowers Law Enforcement
Mitch Beard and Javier Campanini, Palantir

Palantir engineers explain the technology that powers Palantir law enforcement deployments. Learn how police agencies leverage Palantir Gotham for emergency call response, criminal analysis of large data, and case management.


Pursuing the Mission While Protecting Privacy: Not a Zero-Sum Game
John Grant and Bryan Cunningham, Palantir

This presentation discusses how Palantir’s baked in privacy and civil liberties protective capabilities can be used to facilitate compliance with data protection requirements common throughout the U.K. and Europe. These capabilities can be useful both to the government agencies analysing the data and the private sector service providers retaining and sharing the data, and they can be applied to the data protection needs of any Palantir customer. Palantir’s Privacy and Civil Liberties team uses the proposed Communications Data bill as context for a discussion of how Palantir Gotham supports rigorous data protection policies.

We’ve been working with the National Center for Missing and Exploited Children for three years now, and their tireless dedication to protecting children continues to inspire us every day. We recently made a short video that tells the story of our partnership. NCMEC analysts describe how Palantir Gotham helps them make sense of all their data faster, so they can track down missing children quickly and stay ahead of predators.

NCMEC is one of the partner organizations that our Philanthropy Engineering team supports through donations of our technology and technical expertise. We encourage everyone to check out NCMEC’s new blog and donate to this worthy organization.

When we first started developing our data fusion platforms, we wanted to create a way to integrate data for secure, collaborative analysis at scale. We knew that achieving this outcome would not be a matter of simply making more data available to more analysts. We wanted to give our users a way to learn from and build on not just the raw data in their enterprise, but the insights of their colleagues, too. The right technology had to integrate both data and analysis.

As we grew and deployed our technology to more (and more diverse) users, we saw first-hand how collaborative data analysis can save time, save money, and even save lives. We also saw a lot of unrealized potential. If the advantages were so apparent, we wondered, why didn’t we observe more collaborative data analysis between organizations where policy allowed it—allied nations, neighboring public institutions, and partner commercial enterprises?

In turns out that integrating data for secure, collaborative analysis isn’t as easy as throwing everything onto a share drive. Many of our customers wanted to collaborate more, but were prevented from doing so by several hard, multi-faceted technical problems:

• How do you resolve conflicts that might arise from an act of data sharing? If duplicate or conflicting versions of the data are created, there is a risk of obsolete or inaccurate data being spread throughout the enterprise. Alternatively, if data is overwritten, valuable knowledge risks being destroyed.
• How do you collaborate in situations where analysts are geographically dispersed, and communications between them are plagued by intermittent connectivity, high latency, or low bandwidth?
• How do institutions collaborate when they have different missions and different ways of modeling their world? How does a regulatory agency collaborate with a global financial institution in a way that allows each to retain its own data model?
• How do you implement the right privacy safeguards when sharing data both within and between different organizations? How do you protect data at a granular level, so it can only be accessed by those who are authorized to do so?
• How do you enable data sharing without compromising data security when different organizations, and different data sources within organizations, are subject to different data protection and retention policies, classification levels, or access control regimes?

Nexus Peering is Palantir’s solution to these problems. Nexus Peering enables information-sharing at the institutional level, allowing teams, agencies, and governments to exchange data and analysis in almost any direction or environment while maintaining consistency, integrity, and security.

Understanding Nexus Peering

So how does Nexus Peering work? A comprehensive explanation would require more than a simple blog post. But for introductory purposes, you can think of each installation or instance of our Palantir Gotham data fusion platform as maintaining its own “nexus” of data. Nexuses can incorporate changes made by users at different Palantir Gotham instances through acts of synchronization, or “peering.” The main goal of Nexus Peering is to ensure that data is always in a consistent state across instances. To this end, Nexus Peering must capture, circulate, and merge changes to shared data while recognizing and resolving data conflicts.

Each nexus maintains a record of each change to every piece of its data in a manner similar to how revision control software such as Git or Mercurial tracks changes to codebases during software development. Upon log-in, each user “checks out” a copy of the consensus view of the data (known as the Base Realm) and works with it on a private branch. When a user wishes to share the results of his or her analysis with the rest of the team, he or she publishes the changes back to the Base Realm.

Nexus Peering keeps track of these changes across nexuses, even in cases where the peered instances are not in constant communication. In order to merge changes that were made to the data concurrently by users at different nexuses, Nexus Peering uses a technique called ‘version vectors’ to keep track of and automatically apply the changes in the proper order when peering. In cases when concurrent changes conflict, users are alerted and forced to determine for themselves which version of the data they want to work with.

That’s a brief overview of how Nexus Peering works. For an in-depth explanation, check out the presentation from GovCon 6 below.

Nexus Peering in Palantir Gotham 3.8 And Beyond

Late in 2012, we upgraded the instances on our largest Nexus Peering mesh network (spanning 40 locations and four continents) to Palantir Gotham 3.8, which includes a host of new Nexus Peering features. Here are some highlights:

• User Attribution: Peered objects now include a rich attribution of each change made prior to peering, indicating which user on the originating instance made the change (and at what time).
• Incremental Peering at Scale: Transferring data “chunks” via Nexus Peering was previously an all-or-nothing process. In 3.8, we have made peering incremental, which is a vital improvement in bandwidth-constrained and massive data scale environments.
• Graph Peering: Users can now publish analysis in the form of finished graphs for peering to other systems.
• Cross-Domain Peering: Where appropriate, Nexus Peering can now be performed across different networks. Customers can move data from lower to higher classification levels through existing one-way data guards, even if it requires re-writing all relevant data in human-readable form. Where allowed, a Palantir customer with classified data can now peer all data at a lower classification on an ongoing basis, whether this data is their own or a partner agency’s.
• Cross-Classification Peering: Systems with different classification levels (for example, friendly agencies of different allied nations) can now peer with each other. Nexus Peering allows administrators to define classification translations that safely map one set of classifications onto another.

Today, Nexus Peering works across data models, security models, time zones, and borders. It does not require all data to be shared and it does not require an active connection between the two nexuses being peered. Nexus Peering captures and circulates changes between nexuses, while recognizing and resolving conflicting changes, all without destroying or duplicating any data. With Nexus Peering, institutions can create a unified, shared, enterprise-wide information picture that is updated at whatever frequency is possible. Sometimes that’s as real-time as a high-speed network will allow, sometimes it’s as fast as the physical drive containing the latest updates can be hand-delivered.

This is what a complex Nexus Peering network might look like. Each box represents a Palantir Gotham instance, or “nexus.” (Place names chosen at random from a Diplomacy board.)

Based on the technical improvements introduced in the latest version, we envision even more compelling use cases for Nexus Peering. Cooperation between different organizations, including international partners, is now significantly easier, enabling unprecedented secure collaboration on common threats such as cyber attacks. The same security frameworks that allow governments to share data can now also allow secure sharing between the private sector and government, while protecting individual privacy and organizational sensitivities. New use cases could include banks sharing cyber threat data with law enforcement, pharmaceutical companies sharing product safety data with regulatory agencies, peering between agencies involved in disease control, and peering between our philanthropic partners. We’re looking forward to seeing what the future of Nexus Peering holds.