Last month we hosted Palantir Sync: Investigate, which brought together members of the investigative, regulatory, and prosecutorial communities for a discussion of how technology and data are transforming their missions. Through a series of presentations we demonstrated how our software could provide a single, end-to-end platform for their investigations—from initial lead all the way to prosecution.

Investigate was held at the historic Carnegie Library in Washington, D.C., which presented a stark contrast to the technologies on display. Keynote speaker Patrick Fitzgerald, former US Attorney for the Northern District of Illinois and current partner at Skadden, Arps in Chicago, opened with a keynote speech that walked the audience through the evolving role of technology in investigations and prosecutions over the past 20 years. He described the state of the world 20 years ago, when running a query on a computer was considered reckless. Fitzgerald spoke to the dramatic shifts in process, practice, and perspective that have now led to greater information sharing between agencies. a shift facilitated by a technology like Palantir’s that allows for integration of a vast amount of data, of different types and sources, audit trailing, and robust access controls.

The Evolving Role of Technology in the work of Leading Investigators and Prosecutors

Patrick Fitzgerald | Partner | Skadden, Arps, Meagher & Flom | Former United States Attorney for the Northern District of Illinois

After highlighting some of the critical components of Palantir, Fitzgerald said,

“I don’t know how in the world you could go after important targets without a very sophisticated system like that.  On the private sector side, going up against the government without that on your side as well is like bringing a knife to a gun fight.”

Fitzgerald’s presentation was followed by a series of Sync Sessions, which provided deeper looks at how Palantir’s software is currently used across local, state, and federal enterprises to pursue criminal and civil cases from lead, to arrest, to prosecution. The sessions included a demonstration of how U.S. Immigration and Customs Enforcement is using our software to investigate human trafficking. For the first time, Palantir Mobile was demo’d on the main stage to showcase field operation work with the New York City Mayor’s Office of Special Enforcement. Ryan Taylor, Legal Counsel at Palantir, closed the Sync Sessions by illustrating how Palantir Gotham has been used in criminal and civil cases with federal, state, and local prosecutors.

Check out the Sync: Investigate sessions below

Welcome and Introduction
Alex Karp | CEO and Founder, Palantir Technologies

Building a Human Trafficking Case from A Lead to Arrests
Timothy Wargo | Unit Chief | Information Sharing and Infrastructure Management | Homeland Security Investigations | Immigration and Customs Enforcement
Ryan Beiermeister | Forward Deployed Engineer | Palantir

Palantir’s work with the U.S. Immigration and Customs Enforcement began over two years ago as part of Operation Fallen Hero, which was ICE’s response to two of their agents being ambushed on an operation in Mexico. Today, ICE uses Palantir’s software across the enterprise for case management and investigative work, including human trafficking cases. Following an introduction to our work with ICE, Ryan Beiermeister demonstrates how Palantir Gotham is used to build a human trafficking case.

Using a Mobile Date Platform for Field Operations
Kathleen McGee | Director | New York City Mayor’s Office of Special Enforcement
Logan Rhyne | Forward Deployed Engineer | Palantir

The New York City Mayor’s Office of Special Enforcement is responsible for coordinating efforts across City’s agencies to investigate and solve quality-of-life issues city-wide, including lawless clubs and counterfeit trademark activity. The Office conducts investigations in the field in response to complaints, handles the subsequent mid-level litigation, and develops policy to address the city’s issues. See how Palantir Gotham is used as the Office’s platform for data analysis and how Palantir Mobile is used to collect data and execute operations in the field.

Prosecuting Big Data Cases from Insider Trading to Healthcare Fraud
Ryan Taylor | Legal Counsel | Palantir

Learn how Palantir Gotham has helped prosecutors overcome a variety of data integration and analytic challenges to prosecute cases involving vast amounts of data. Ryan Taylor, Palantir’s Legal Counsel, walks through how the Gotham platform was leveraged in a number of real criminal and civil cases, including insider trading, money laundering, insider trading, and health fraud.

Our Philanthropy Engineering team is proud to support a number of non-profits that are working to prevent atrocities. Our partners are contributing to real-time early warning and prevention of violence against civilians, using open source data-driven analysis to identify those financing and otherwise enabling atrocities, and helping to tell the story of mass atrocities occurring around the world. We have learned a lot from these partnerships about the role that technology can play in preventing atrocities, and are always looking for opportunities to share these lessons with like-minded organizations.

On May 27^th and 28^th, I traveled to Arusha, Tanzania to attend “Best Practices and New Opportunities in Genocide Prevention: Governmental Action, Technology, and Regional Contexts,” a conference organized by the Auschwitz Institute for Peace and Reconciliation (AIPR) in conjunction with the Tanzanian Ministry of Constitutional and Legal Affairs and the United Nations Office of the Special Advisor on Genocide Prevention and the Responsibility to Protect. On a panel dedicated to exploring how technology can support the effective response to and prevention of mass atrocities, I presented the work of two of our partner projects: Resolve and Invisible Children’s LRA Crisis Tracker and the Center for Advanced Defense Studies’ (C4ADS) work on illicit trade and its relation to atrocities.

Photo Credit: Auschwitz Institute of Peace and Reconciliation

The LRA Crisis Tracker uses Palantir Gotham to track the activity of Joseph Kony’s Lord’s Resistance Army in Central Africa. Individuals report LRA attacks via early warning systems that have been put in place by Resolve and Invisible Children. These organizations then use this data to respond to atrocities and spread messages encouraging LRA members to defect.

Photo Credit: Auschwitz Institute of Peace and Reconciliation

At the Arusha conference, I demonstrated how analysts can analyze patterns in LRA activity not just to decide where and when to institute new defection programs, but also to measure the effectiveness of those actions after the fact—for instance, does a new flier drop program in a certain area lead to any measurable or noticeable increase in defections in that area in the following weeks?

C4ADS, a think-tank based in Washington, DC, has used Palantir Gotham to identify people and organizations that are intentionally or unintentionally enabling atrocities. In Arusha, I explained how a few analysts were able to quickly and comprehensively analyze a web of shell companies operating ships that were caught illegally trafficking Somalian charcoal (so-called ‘conflict charcoal’ because it is used to fund the al-Qaeda linked al-Shabaab terrorist group). Using Palantir Gotham as a search and discovery tool to analyze open source data from the UN, news media, business directories and other sources, C4ADS was able to identify some key individuals and organizations linked to the illicit trade off of the Horn of Africa.

My panel also included some great discussion of the importance of protecting privacy and civil liberties when collecting, aggregating, and analyzing data for atrocity prevention. The data associated with these conflicts is often messy or incomplete, and is typically spread across multiple sources and stored in disparate formats. Palantir and our Philanthropy Engineering team are dedicated to building technology that can integrate this information and render it useful to organizations working to prevent atrocities, while protecting the privacy of those at risk.

Mojo the HeroRAT

The conference was a wonderful experience, and I learned a lot from the other attendees. As I packed to come home, my only regret was that my schedule didn’t allow for a trip down to Morogoro, where another one of our philanthropic partners is based. His name is Mojo, and he’s a rat. Since last September we at Palantir have been proud to support the Belgian NGO APOPO via the adoption of Mojo, a “HeroRAT”. HeroRATs are used by the NGO to sniff out and identify unexploded landmines and tuberculosis. The rats are too light to detonate the landmines, so once they identify one through scent and scratch to indicate its location, humans are deployed to safely remove the mines.  Our HeroRAT Mojo has been in intensive training in Morogoro. Although I didn’t get the chance to visit him on this trip (he was too far from Arusha), we have been keenly following Mojo’s progress through advanced training and his final, rigorous landmine clearing tests. I look forward to meeting him the next time I’m in Tanzania!

Team Pegasus’s quadcopter gathers on-the-scene damage assessments via aerial imagery for better disaster recovery.

Hack Week: a Palantir tradition

Now in its fifth year, Hack Week is an annual tradition at Palantir. During Hack Week, all normal work on our products stop and ad-hoc teams compete to create prototypes of new features (or even whole new products). While innovation is a normal part of work in crafting Palantir’s Data Fusion Platforms and the constellation of technologies that support them, Hack Week is intended to give everyone a blank canvas. People are encouraged to build things that take us in unexpected and unplanned directions. It’s a week of unbound creativity, throttled only by the need to sleep and the looming deadline of the Hack Week Final Presentations.

Team C-Dog presents their work on operational transforms to enable realtime collaboration.

In years past, various Hack Week projects have become full-fledged parts of our product lineup. Notably:

• Palantir Mobile is a mobile client that allows personnel in the field to collaborate in real time with analysts at a central location. Palantir Mobile is now an integral part of a number of our customers’ field operations, including our philanthropic work with Team Rubicon to improve the speed and efficacy of disaster recovery.
• Horizon was a Hack Week project in 2009 that empowers interactive analysis on billions of objects. An in-memory query system that’s integral to many Palantir workflows, it’s similar in architecture to the AMPLab’s more recent Spark project.
• Palantir Gotham’s Map Application which allows analysts to combine semantic and link-relation analysis with a rich set geospatial visualization tools came out of the original Hack Week in 2008.

See our post from last year’s Hack Week for a short video that really captures the energy and spirit of hack week.

Watching the presentations as teams ship what’s done at the end of the week.

Bigger than ever before

Timed to allow our summer interns to participate, over 100 teams formed and got down to work. 88 different teams submitted finished projects for consideration. 12 projects made it to the final stage to place in six different categories.

One project used RaspberryPi and lock sensors to automate the tracking of shower availability in the office.

And the winner is…

This year’s finalists were quite varied in the different problems that they tackled. Here are a few:

• A project that mated a GoPro camera to a lightweight quadcopter, along with all the software integration to fly it and bring in imagery to aid in disaster recovery damage assessments.
• A web-based version of the Palantir Gotham Graph application that allowed multiple users to collaborate in real time.
• A system that uses Elasticsearch on top of Palantir Phoenix to do interactive querying and filtering of very large datasets for cyber security workflows.
• A project to distill a Palantir Metropolis installation such that it can run on a single laptop for smaller workflows and faster setup and prototyping.
• A complete rewrite of the Palantir Metropolis API for web applications to use

Team Bonding Activists shows off their shared real time graph.

Until next year…

Like many hallowed traditions, Hack Week only comes once a year. Until then, we’ll be counting down the days.

Human trafficking is a global problem involving the deception, enslavement, labor and sexual exploitation of vulnerable people. While human traffickers are avid users of technologies like social networking sites to find and target victims, technology is also playing an important role in enabling anti-trafficking organizations to combat the problem. The fight against human trafficking is one of the current priorities of Palantir’s Philanthropic Engineering Team, a team dedicated to solving important problems that face our world through donations of our technology and expertise. On 10 July, our Philanthropy Engineering lead, Jason Payne, participated in a Huffington Post webcast on how technology is being used to combat human trafficking. The webcast featured a panel made up of experts from Palantir, Google, Polaris Project, and CNN.

Anti-human trafficking organizations currently use our software to address the problem in two ways: to respond to victims’ needs in real time, and to analyze the aggregated incident data to understand the reach of trafficking networks. Victims of human trafficking call into hotlines created to help them escape from slavery. Closely watched and controlled, these victims often only have minutes while hiding from their minders to contact the organizations set up to help them. Using our analytical software platform, the anti-human trafficking non-profit organizations Polaris Project, Liberty Asia, and La Strada International can cull data from reported trafficking events, search for nearby service providers, and quickly identify the best way to help based on the caller’s local region – all while the caller is still on the line. These organizations can also map patterns from the aggregated data to reveal trends. For example, analysts can see exactly how trafficking incidents shift with the seasons. Insights like this are exactly what law enforcement officials and policymakers need to address the systemic issues involved in human trafficking.

One of our partners in this effort, Google Giving, donated $3 million dollars to Polaris, Liberty Asia, and La Strada International so that they can expand and develop their existing human trafficking hotline efforts. As Google’s Jacqueline Fuller said about their motivation, “Traffickers are very good at using technology to enslave women and to traffic children, but we need to empower the good guys on the front lines to do the same.”

As with every Palantir project, privacy and civil liberties protections are never just an afterthought. Jason stressed the importance of these protections during the webcast. “We’ve worked very closely with Polaris (as well as the other organizations in this alliance we’re building) to do Privacy Impact Assessments to ensure that if and when data is shared between organizations… that the privacy and civil liberties of these individuals are protected.”

Understanding, tackling and preventing human trafficking requires a global understanding of the problem, the victims, and the traffickers. “We know that trafficking victims are children and adults, men and women. When we think about human trafficking we need to think about everything — including supply chains,” added Margaret Howard, a trafficking survivor herself. In order to get at the whole picture, our technology provides more than just visualizations of the data flowing in from hotlines. Our platform seamlessly fuses all of the data surrounding the problem, while at the same time providing robust, fine-grained access controls to ensure privacy when sharing. While human traffickers increasingly utilize technology to recruit and enslave their victims, we and our partners can leverage technology and data to fight back and make a difference in defeating them.

If you missed it, the webcast can be found here: HuffPost Live: Tech and Human Trafficking.

For more information about human trafficking in the US, to report a tip, or to get help, call the National Human Trafficking Resource Center hotline at 1-888-373-7888 or text BeFree (233733).

In the past year our product design team has added over a dozen amazing designers with a mix of skills spanning interaction design, information architecture, visual design, and user research. Together, we have the privilege of addressing incredibly challenging design problems in support of world-changing missions. Sometimes we focus on refining an existing platform, while at other times we’re leading the exploration of entirely new products and domains. We apply our user-centered design process to many types of problems, from helping detectives re-imagine their criminal cases to recognizing and responding to cyber attacks.

Because we don’t make downloadable consumer products, relatively few people beyond our enterprise customers get to interact directly with our work. This makes it hard for people unfamiliar with Palantir to get a concrete idea of what we do. In the coming months, we’ll be providing more insight into product design at Palantir. We will share thoughts on design thinking, culture, the challenges we face, and the role product designers play in helping our customers solve their most important problems.

In our next post we’ll get into the details of our Code33 design architecture, which is setting the stage for all our web products. Stay tuned!

Dr. Andrew Marx of the US Department of State and a fellow at the US Holocaust Memorial Museum’s Center for the Prevention of Genocide, developed a novel method for analyzing archived data from NASA’s Landsat ETM+ mission to better understand when and where specific villages were destroyed in the Darfur conflict from 2003-08. This study provides the most comprehensive documentation of the geospatial and temporal patterns of villages destroyed in that genocide. This analysis can help determine patterns of coordination between various military forces carrying out attacks or potentially with the Sudanese central government (Sudan’s president, Omar al-Bashir, is wanted by the International Criminal Court for war crimes related to the genocide in Darfur). Understanding these patterns also has the potential to help determine if actions by the international community, such as international condemnations of genocide, had any impact on the rate of attacks.

Palantir has been honored to partner philanthropically with the US Holocaust Memorial Museum and Andrew Marx to document atrocities in Darfur, Sudan. While Dr. Marx developed a novel way of analyzing satellite imagery to determine when villages were destroyed, Palantir was able to help the Museum visualize and understand trends in the destructions that were uncovered. Palantir Gotham’s Map application was used to document what village burnings looked like over time, and the resulting video is available on YouTube (and below):

You can read more about this project on the Museum’s blog at Detecting Human Rights Violations with Satellites: CPG Fellow Proposes a New Approach or download Dr. Marx’s full report, A New Approach to Detecting Mass Human Rights Violations Using Satellite Imagery .

“This overturns centuries of established practices and challenges our most basic understanding of how to make decisions and comprehend reality.” –Big Data: A Revolution That Will Transform How We Live, Work, and Think

The rise of big data has been a rich vein to tap in nearly every discipline and industry. Accompanying this explosion of data and (perhaps more significantly) the computing power to utilize it is a wealth of discussion on the potential effects on nothing less than human understanding and the future of civilization. Latest in the genre is Big Data: A Revolution That Will Transform How We Live, Work, and Think, by Viktor Mayer-Schönberger, professor of Internet governance and regulation at Oxford University, and Kenneth Cukier, data editor at The Economist. The duo offers some familiar cautionary alarm, but is supported by new insight into how data analysis has changed and why it matters.

Mayer-Schönberger and Cukier describe the “datafication” of society – taking something that has never before been treated as data and turning it into a numerically quantified format (they point to research at the Advanced Institute of Industrial Technology in Tokyo that uses sensors to identify how a person sits).  Rather than focus on how datafication will affect any particular field, Mayer-Schönberger and Cukier consider how it will affect the fundamentals of human thought and decision-making, and how that in turn might affect approaches to privacy and individual liberty.

The book outlines three major “shifts of mindset”:

• Analysts are now more likely to be able to collect and analyze an entire data set (or a significant portion of it) instead of seeking out statistically representative portions of a data set.
• Analysts can now tolerate more errors in the data. Because the size of data sets has increased so much, a few errors are far less likely to skew and undermine the entire analysis. Precision may be sacrificed in order to identify general trends.
• Analysts – and decision-makers – may now care less about causation than they do about correlation. Rather than generating and testing a hypothesis by examining the relevant facts, analysts can use big data to answer questions with brute force, plowing through hundreds of millions of mathematical models until a correlation is discovered.

If the last point is true, it means the Scientific Method has been be flipped on its head. It means considering fundamental changes to how humans answer questions and solve problems.  “Society will face a great temptation to… shift to managing risks, that is, to basing decisions about people on assessments of possibilities and likelihoods of potential outcomes,” write Mayer-Schönberger and Cukier, “Big data presents a strong invitation to predict which people are likely to commit crimes and subject them to special treatment, scrutinizing them over and over in the name of risk reduction.”

Furthermore, they worry that as projections become more prevalent and accurate, society may punish people for predicted behavior, negating free will. “The future must remain something that we can shape to our own design,” they write. “If it does not, big data will have perverted the very essence of humanity: rational thought and free choice.”

It’s tempting to dismiss Minority Report-style prophecies as exaggeration, but we’re already seeing shifts in how individuals think and act as a result of vast quantities of readily available data. Just a decade ago, knowledge of youthful indiscretions such as drinking too much at a college party would have been confined to a small circle of friends and perhaps a couple of blurry photos. Today, such activities may be broadcast to hundreds or even thousands of individuals well beyond the college campus, and future employers may make hiring decisions based on this behavior. The digital age has made more aspects of our lives instantly accessible and virtually unforgettable. Consequently, individuals must dramatically adjust their risk calculations before they put that lampshade on their heads at a party, or – more soberly – attend a protest rally in support of a controversial cause.

But agonizing about this particular dystopian future devoid of free will misses the simpler point: big data analysis can have a profound effect on our basic interaction with the world. Sometimes the effects are positive – for example, eliminating certain cognitive biases, or correcting our tendency to see patterns or causalities where none exist. In other cases, these changes can be negative – sapping our curiosity to understand why something occurs, or stifling innovation and inspiration that might “go beyond the data.” It’s the understanding of these analytical shifts as ultimately helpful or harmful that will define the ongoing data revolution.

Companies will need to be ahead of these changes as they design and build new technologies. Palantir has always put a great deal of stock in the role of humans in data analytics. We design capabilities that enhance the ability of a human analyst to understand data and derive actionable intelligence from it; we never displace the human as the ultimate data processor. Even so, design itself affects users – intentionally or not – and influences behaviors based on how the user interface is built and data is presented. Any responsible player in the big data game must wrestle with the question: are we enhancing some human ingenuity at the cost of stifling other unique aspects of our intelligence?

Making this determination is not easy. We can point to enormous success in facilitating criminal investigations for our law enforcement customers. Police officers using Palantir can assess swaths of information deftly, allowing them to identify new suspects or lines of inquiry that might have never surfaced in the past. But it’s harder to know how this powerful tool is changing how officers think in their day-to-day work. Are officers ignoring their instincts – sometimes honed by years of police work in the field – in favor of what the data is telling them? Would those instincts have led them up an investigative blind alley or might they have resulted in the sort of intuitive mental leap that can crack a case wide open – and that no computer could ever make?

These questions are why Palantir engineers spend so much time in the field working directly with customers and analysts to understand how they did their jobs before Palantir. We want to preserve what worked while addressing friction points that hampered success. It’s why the development of new features in Palantir is largely user driven rather than dictated by the assumptions of engineers. Mayer-Schönberg and Cukier’s book reinforces the urgency of our mission: to provide technology that preserves the essential role of human judgment and individual responsibility in big data analytics, and to do so through design and engineering practices that augment sound decision making rather than artificially shifting or displacing that which already works.

This week marks the one year anniversary of Superstorm Sandy’s landfall on the eastern seaboard of the United States. It’s an occasion to remember those who lost their lives to the storm, and to be reminded of how much work remains to be done to rebuild. It’s also an occasion to thank again the thousands of volunteers who flocked to affected areas and provided relief to many of the victims in the immediate aftermath of the storm.

We were privileged to work alongside our philanthropic partners Team Rubicon and Direct Relief and their many volunteers to coordinate disaster response operations after Sandy. It was our first disaster response deployment, and it quickly showed us how our technology could make a big difference in this area. Just a few months after we arrived in Rockaway Beach, in conjunction with Team Rubicon and Direct Relief, we made a Clinton Global Initiative (CGI) Commitment to Action to expand our efforts. Since February, Team Rubicon and Direct Relief have been using our technology to provide relief to those affected by natural disasters across the country, including the victims of the tornadoes that struck Moore, OK in May and the floods that hit Colorado in September.

To mark this occasion, we prepared a short video in partnership with CGI. Together, we’ve accomplished a lot in the last year and we look forward to doing even more in the years ahead.

C4ADS has been in the spotlight lately. A front-page story in a recent Sunday edition of the Washington Post highlighted the D.C.-based organization’s data-driven research into illicit networks trafficking weapons and other goods throughout various conflict zones. Over the past year, their research into enablers of conflict has led them on a path of analysis from illegal shipments of ‘conflict charcoal’ out of Somalia to commercial ships that may be carrying arms and other sensitive cargo into Syria. In September, C4ADS released a report on a group they called the “Odessa Network,” claiming that this network had been transporting arms to the Syrian government via Ukraine. And as Senior Analyst Tom Wallace put it, C4ADS “couldn’t have done it without Palantir.”

Using the Palantir Gotham platform, C4ADS has been able to bolster their mission of using data to better understand conflict, security and development around the world. This latest analysis relied on Palantir’s ability to combine multiple, disparate open source datasets to reveal a complicated network of ships, transport companies, and key individuals involved in the transactions.

C4ADS’ investigations combined information from news media, court cases, open source business directories, maritime shipping databases, and ship transponder records. These diverse datasets arrived in various formats, and without a unifying analysis layer, they would have provided only a small piece of the puzzle. C4ADS therefore found Palantir’s data fusion platform “transformational” in improving the efficiency of their analysts.

C4ADS recently discussed their findings on the Odessa Network with the New America Foundation, and their work was highlighted at “Conflict in a Connected World,” the October 2013 summit presented by Google Ideas.

Palantir’s software allowed C4ADS analysts to “tease out the important needles in these messy, multi-faceted haystacks” of data, said Wallace, making it possible to “keep track of how we know what we know.”

Typhoon Haiyan has claimed the lives of thousands and displaced millions more. Along with other aid organizations from around the world, our disaster response partners Team Rubicon and Direct Relief have mobilized to provide relief to those affected by the storm, and we’ve been working closely with them to support their efforts.

On Veterans Day Team Rubicon launched Operation: Seabird, which sent a specialized team of veteran volunteers to Tacloban. Equipped with kits supplied by Direct Relief, they’ve been conducting search and rescue, patient extraction, and medical relief operations. Second and third teams have deployed more recently to expand operations and prepare for the arrival of even more Team Rubicon volunteers in the days ahead.

Direct Relief has sent several emergency airlifts of medical supplies and medications to affected areas, and have now made their entire $65 million medical inventory available for the relief effort. They will soon be deploying personnel of their own to assess local health needs directly.

We’ve been hacking away furiously all week to support these efforts. Here’s an update of what we’re already doing and what we have planned.

Raven with live data from Tacloban.

Putting data (and our engineers) to work

We quickly stood up a Palantir instance and are actively integrating and structuring OCHA reports, MapAction updates, various open source data sets relevant to the area of operations (OpenStreetMap, local government info, supply centers, shelters, markets, etc.), SITREPs from various relief organizations, assessment reports generated in the field, and more. By providing access to many different kinds of data, the most recent data, and helpful ways of interacting with that data, we hope to help Team Rubicon and Direct Relief better coordinate their actions with other NGOs and provide more relief to more people, faster.

On the UX front, we were especially excited to deploy Raven, our new high-performance, web-based geospatial analysis application. We’ve loaded it with a range of static and dynamic map layers, and we’ll be using it for all things geospatial—from GPS tracking of volunteers (see more below) to operational planning, visualizing the distribution of resources, and more. Thanks to DigitalGlobe, Raven has access to near real-time satellite imagery and crowd-sourced damage reports. The Raven dev team has been working day and night all week to build out new features specifically for this mission.

The instance is being manned by a team of Palantir engineers as well as Direct Relief and Team Rubicon users. Stateside, we will be providing various kinds of reachback support—integrating data, responding to RFIs, cleaning up and enriching new data as it comes in, and generally making sure the right people know what they need to know when they need to know it. Our users in the Philippines will depend on the instance to plan and execute their day-to-day operations.

Building a tech package for disconnected environments

We recently confirmed that Team Rubicon can connect to the main instance from Tacloban, but we expect there will be areas where we can’t get that kind of connectivity, so we’ve prepared a technical package that will allow the team to operate in a completely disconnected environment.

Moving Palantir Forward, forward

Several years ago we developed a standalone version of Palantir that could run on a laptop in disconnected environments such as conflict zones. We called it Palantir Forward. It allows users to take a cut of data out to a remote location, perform analysis and add new data while disconnected, and then sync that data with other Palantir instances when connectivity is restored. We sent nine Palantir Forward laptops to the Philippines. Now the same technology that was built for active duty soldiers will be used to help veterans provide disaster relief.

Deploying Palantir Mobile, and introducing Mimosa

Palantir Mobile was one of the most effective capabilities we deployed after Hurricane Sandy. It enabled Team Rubicon volunteers to submit damage assessment reports from the field back to the FOB in an instant, and it allowed planners at the FOB to track the location and movement of their volunteers. This removed the latency associated with hand-delivering reports, removed the overhead and risk associated with managing paper-based reports, and streamlined comms across the strategic and tactical levels of the operation. (You can watch a short video about how Team Rubicon used Palantir Mobile after Sandy here). We think a similar capability could be beneficial to Haiyan relief efforts.

Team Rubicon brought Palantir Mobile handsets with them to Tacloban, but these devices depend on cellular connectivity. We may not be able to depend on cellular network infrastructure in some of the most damaged and remote areas. So we spent the last week coming up with a purely satellite-based version of Palantir Mobile. We’re provisionally calling the result the MIni MObile SAtellite, or Mimosa. (Admittedly, not a great name. I blame the sleep deprivation.)

A successful test of Mimosa.

Thanks to a days-old partnership with Delorme (and their great APIs, engineers, and client service people), we have prototyped and successfully tested a capability that works with inReach two-way satellite communicator devices. Direct Relief purchased 130 inReach devices, which are equipped for GPS tracking anywhere in the world and can send and receive 160-character, Tweet-like messages over satellite comms. Our Raven dev team built a plugin that will allow us to use the GPS data for real-time Blue Force Tracking of volunteers in the field. Message data will also show up on the map, so users at base stations will be able to communicate directly with forward teams from within the Raven interface. We’ve written some data integration scripts against the inReach API to generate data objects based on situation or assessment reports sent from these devices. To accommodate the character limit, we hacked together a shorthand system so the data can be easily sorted and analyzed in Palantir. Here’s an example of how it works. If someone were to send the following message:

SAR–60–0–20-Water-/-

it would be interpreted by Palantir as:

Zone Search-and-Rescue Report: 60 percent of homes habitable, utilities not intact, estimate 20 deaths, water needed, no other details.

Once volunteers start submitting reports from the field, whether via Palantir Mobile or Mimosa, we will quickly get a better picture of where resources need to be directed to deliver relief to those who are most at risk.

Deploying engineers to the Philippines

Over the weekend a small team of Palantir engineers from our California, Virginia, and Australia offices arrived in the Philippines to support relief efforts on the ground. The team includes engineers who have previously deployed to conflict zones, a Tagalog speaker, and a Palantir Mobile specialist. They’re geared up for harsh conditions, unpredictable technical challenges, and the chance to work with our amazing partners to make a real difference.

How you can learn more

We will continue to publish updates as we get more information. To stay up to date on developments, follow the whole team on Twitter @PalantirTech, @TeamRubicon, and @DirectRelief, and check out the latest news on the Team Rubicon and Direct Relief blogs.

How you can help

Support the response efforts by making donations to Direct Refief and Team Rubicon. Arm volunteers on the ground with better data by participating in DigitalGlobe’s crowdsourcing campaign to tag damage in satellite imagery.

Although Palantir is mostly known as a Java Swing shop, we have quietly (or not so quietly) been building for the web for a while now. From D3 for stunning graphics to Backbone and Angular for large-scale applications, we are using the latest and greatest web technologies to build the next generation of our products. As we’ve made the transition from native to web applications, we’ve tried to preserve some features of the strong development process that has served us so well in the native world: robust unit and end-to-end testing, continuous integration builds, and most importantly, a developer experience that makes it easy and fun to write the data analysis platforms and applications that our customers rely on to solve their most important, most complex data problems. We found TypeScript’s optional typing and forward-looking adoption of certain ECMAScript 6 features to be a great fit for those needs, but we wanted a bit more tooling. So today we’re pleased to announce three new open source projects we’ve undertaken to help fill out the TypeScript ecosystem.

Eclipse TypeScript plug-in

While we love Sublime Text at Palantir, we thought it would also be cool to have more of an IDE experience for TypeScript (probably because it is something we are familiar with from Java). Although there was already excellent support via WebStorm, we really wanted to have a first-class experience in Eclipse as well since it’s our primary development environment for our server code. We knew this would be a pretty involved task given all the functionality offered by a modern IDE, but one of our summer interns, Tyler Adams, bravely took on the project. The TypeScript team has done a great job of exposing a lot of the basic services necessary to build a robust IDE experience: auto-completion, syntax highlighting, code compilation, etc. All we had to do was figure out how to hook up these great services into Eclipse via a plug-in. One of the fun aspects of this project was that since parts of the plug-in are written in TypeScript, we were really motivated to get features done to make writing the plug-in itself easier. By the end of the summer we got pretty far (check out the feature list) and were so happy with the plug-in that we converted all of our code over to TypeScript (more about this below). There is always room for more improvement though, so we’d love to have your suggestions and code contributions. You can check out the project here: https://github.com/palantir/eclipse-typescript.

TypeScript auto-complete in Eclipse.

TSLint

Our backend code is mostly written in Java and we love tools like FindBugs and Checkstyle for ensuring that coding best practices are followed and common pitfalls are avoided. One of our engineers in NYC, Ashwin Ramaswamy, wanted to have some of those same assurances for our TypeScript code. He looked into using existing tools such as JSHint or ESLint but decided that we really needed something that would work directly on the TypeScript code. Each summer, we throw a Hack Week which is a week during which all normal work stops and everyone works in teams on fun projects they would like to see come to fruition. This past summer, Ashwin chose to create TSLint as his project. He didn’t just stop with the linter though – there are also plug-ins for Eclipse and Grunt to make it easier to use. Check it out at: https://github.com/palantir/tslint.

CoffeeScript to TypeScript converter

We use CoffeeScript for most of our projects at Palantir but we found that for some really large ones, optional typing can be a real boon to productivity. We learned this lesson the hard way after trying to maintain a medium-sized application and finding that we had to write an incredible amount of unit tests to be certain that even the simplest of edits (like renaming a variable) were safe. This summer, we asked one of our interns, Jared Pochtar, to work on an automated method of converting our CoffeeScript code to TypeScript. He came up with the brilliant idea to slightly modify the CoffeeScript compiler to output TypeScript instead of JavaScript. If you’d like to try out converting your codebase to TypeScript as well you can check out his work here: https://github.com/palantir/coffeescript-to-typescript.

With the recent release of TypeScript 0.9.5, we’ve begun to see the path towards 1.0 really take shape. We think TypeScript is going to be a great addition to the web community. We are really looking forward to see where it goes next and what people do with it!

We officially welcomed our long-time partner The National Center for Missing & Exploited Children (NCMEC) to Palo Alto last Thursday. Since 1984, NCMEC has worked tirelessly to protect children throughout the country from abduction and exploitation. It is a mission that we are extremely proud to support by donating  our software and engineering expertise.

Though NCMEC maintains a regional southern California office, a local presence in Silicon Valley will further boost their efforts to work closely with the technology industry. A formidable technological response is required to stay ahead of child predators and track down missing and exploited children. Palantir has been assisting NCMEC in this regard since 2010, and their new proximity will make this partnership even more beneficial and productive.

NCMEC’s new Palo Alto office.

With Palantir, NCMEC analysts can make sense of large volumes of dispersed data and draw connections to help law enforcement fight the abduction and sexual exploitation of children. Sources such as internal NCMEC databases, case reports, public records, open source data, websites, social media, maps, images, videos, can all contribute clues that can help bring children to safety. By integrating these databases into one place, analysts can search and investigate the data in minutes or seconds. This frees up time to do more work and provide greater assistance to law enforcement. Learn more about how NCMEC is using Palantir in the video below.

NCMEC’s new location will help yield more success stories, like the case of a 17-year old girl who was reported missing and potentially involved in child sex trafficking. Through various searches, a NCMEC analyst was able to find multiple posts online that advertised this missing child for sex. Through information in the ads, the analyst was able to tie them to other posts from the same pimp. The analysis included more than 50 advertisements, 9 different females, and a trail that covered 5 different states. A link analysis graph was created using Palantir that allowed law enforcement to easily see the large scope of the ring. This insight helped law enforcement link the pimp to a multitude of other crimes and other girls that he victimized.

With NCMEC now firmly planted in Palo Alto, we encourage other tech firms to support NCMEC’s life-saving work. We are incredibly proud and inspired to work with NCMEC, and welcome them to the Valley.

Each year, human traffickers reap an estimated $32 billion in profits from the enslavement of 21 million people worldwide. And yet, for most of us, modern slavery remains invisible. Its victims, many of them living in the shadows of our own communities, pass by unnoticed. Polaris Project, which has been working to end modern slavery for over a decade, recently released a report on trafficking trends in the U.S. that draws on five years of its data. The conclusion? Modern slavery is rampant in our communities.

January is National Slavery and Human Trafficking Prevention Month, and President Obama has called upon “businesses, national and community organizations, faith-based groups, families, and all Americans to recognize the vital role we can play in ending all forms of slavery.” The Polaris Project report, Human Trafficking Trends in the United States, reveals insights into how anti-trafficking organizations can fight back against this global tragedy.

For the past year our Philanthropy Engineering team has partnered with Polaris Project to provide them with our software and engineering expertise. In addition to serving victims and advocating for anti-trafficking public policy, Polaris Project operates the National Human Trafficking Resource Center Hotline (NHTRC). Victims and witnesses of human trafficking can call the hotline to report tips, request help, and connect with anti-trafficking services. Palantir has been instrumental in helping victims and callers quickly access the resources and help they need.

Polaris Project CEO Bradley Myles recently sat down with us to discuss how our software enables Polaris to do more than respond to individual calls to the hotline by discovering connections between cases and identifying global trafficking patterns and networks.

Footage of Polaris Project courtesy of William Caballero.

Polaris Project uses Palantir Gotham to leverage the data from nearly 100,000 calls. NHTRC may collect up to 170 different quantitative and qualitative variables per case record. These data originate from disparate sources—calls, emails, SMS, online tip reports, and publicly available information about trafficking. By integrating this data into a single platform, along with their national referral database of 3000 contacts that includes anti-trafficking organizations, legal service providers, shelters, coalitions, task forces, law enforcement, and social service agencies, Polaris can locate emergency response resources and identify critical services for victims of trafficking in a matter of seconds.

In September 2013, we were honored to host Bradley Myles at Palantir Night Live, where he described the full scale and scope of human trafficking worldwide and shared his thoughts on how technology can help eradicate this crime once and for all.

We are proud to work with Polaris Project to help bring this issue back into the light, raise awareness, and combat the problem wherever it appears.

In 2012, our PCL team assembled the Palantir Council of Advisors on Privacy and Civil Liberties (PCAP), a body of experts in the privacy and civil liberties field who help us understand and address the complex privacy and civil liberties issues that arise in the course of building a sophisticated data analytics platform. The group continues to meet on a regular basis to discuss an ever-growing array of topics and provide invaluable advice to assist Palantir in enhancing the privacy and civil liberties protections built into our powerful analytic platforms.

In light of our growth in international markets and the general globalization of privacy, civil liberties, data protection, and human rights issues, the PCAP has decided to expand to add four new members who will bring more international expertise to the discussions. We are pleased to welcome the following new members to the PCAP:

• Alex Deane – Currently the Head of Public Affairs at Weber Shandwick, Alex was also the founding Director of Big Brother Watch, a prominent U.K. civil liberties advocacy organization.
• Sylvain Metille – Head of the Technology and Privacy practice at BCCC Attorneys-at-law LLC in Switzerland, where he specializes in data protection and surveillance issues.
• Omer Tene – Vice President of Research and Education at the International Association of Privacy Professionals, Managing Director of Tene & Associates, and Deputy Dean of the College of Management School of Law, Rishon Le Zion, Israel, Omer is also a senior fellow at the Future of Privacy Forum.
• Nico van Eijk – Professor of Media and Telecommunications Law and the Director of the Institute for Information Law at the University of Amsterdam, Nico has written extensively on a number of highly technical topics.

The PCAP is looking forward to interesting new perspectives and valuable contributions from our new members.

Virtual labs where doctors and prosthetic designers can collaborate in the same “room.” Touch screen interfaces for cars. Robots that work with autistic children. Sequencing the genome of cancer cells.

These are just some of the projects that the 2013 Palantir Scholarship for Women in Engineering finalists are working on. These nine women were invited to visit Palantir’s Palo Alto headquarters on February 28 for a full day of demonstrations, interviews, panel discussions, and a tour. The event was meant to offer a taste of what a modern, collaborative working environment looks like, as well as a sampling of the kinds of issues involved in Palantir’s work. After a long day of conversation and discovery, the finalists and their Palantirian hosts were inspired by the variety of meaningful work being pursued by women in computer science and STEM fields.

The finalists in a session with Angela Muller

Palantir initially established the scholarship in 2012 to support underrepresented populations in computer science. This year the scholarship extended eligibility to women in all the STEM disciplines, from computer science to engineering, life sciences, and more. The goal behind this expansion was to provide opportunities not only to potential developers, but to women in all technical fields who might build or rely on technology like Palantir’s in their labs or elsewhere in their work.

Eugenia Gabrielova

With this broader perspective in mind, the scholarship essay prompt this year asked applicants to choose a data set and describe how they might analyze it to find insights into how best to promote opportunities for women in technology. This year’s winner, Eugenia Gabrielova, suggested using the GitHub Archive data to analyze patterns of women’s contributions to software projects and to build mentorship networks. A third year Ph.D. student in Information and Computer Science at U.C. Irvine, Gabrielova is no stranger to the challenges scoped out in her proposal – she was the only woman in her graduating class at Northwestern to receive an engineering degree in Computer Science. She credits faculty members for treating her as an equal, not a special case. “They never pushed me to ‘represent ladies in C.S.,’” she said, “They met me where I was.”

Each of the finalists approached the scholarship with unique academic and professional interests, but they all shared a passion for supporting and nurturing women’s contributions in technology. Meena Boppana, an undergraduate at Harvard, started a math club for girls in low-income charter middle schools in response to the frustration she felt as the only girl on her own high school math team. Grace Gee, also at Harvard, is starting a national college initiative to help recognize and reward women involved in technology. She hopes her initiative will do for others what a similar program did for her as a high school student looking for ways to explore opportunities outside of her small hometown in southeast Texas.

The Palantir Scholarship for Women in Engineering is just one small part of a larger, growing movement to support women in technology. This movement stands to benefit not just individual women and their careers, but their professional and academic communities as well. “By excluding women, we’re missing a huge brain trust,” said Henny Admoni, a finalist and first-year Ph.D. at Yale University. Palantir is therefore pleased to congratulate all of its finalists on their achievement and commitment to their respective communities.

• Eugenia Gabrielova - Ph.D. candidate, Information and Computer Science; University of California-Irvine
• Meena Boppana –  A.B. candidate, Computer Science; Harvard University
• Grace Gee – S.M. candidate, Computational Science and Engineering; A.B. candidate, Computer Science; Harvard University
• Annie Liu – Ph.D. candidate, Computer Science; Princeton University
• Henny Admoni – Ph.D. candidate, Computer Science; Yale University
• Edna Sanchez  - B.A.S. candidate, Computer Engineering; University of British Columbia
• Preeti Bhargava - Ph.D. candidate, Computer Science; University of Maryland
• Tracy Ballinger - Ph.D. candidate, Bioinformatics; University of California-Santa Cruz
• Kayo Teramoto – B.S. candidate, Electrical Engineering and Computer Science; Yale University

The 2013 Finalists. Back row, left to right: Kayo Teramoto, Annie Liu, Tracy Ballinger, Henny Admoni, Edna Sanchez; Front row, left to right: Grace Gee, Preeti Bhargava, Eugenia Gabrielova