How the cops watch your tweets in real-time
Recent leaks about the NSA’s Internet spy programs have sparked renewed interest in government surveillance, though the leaks touch largely on a single form of such surveillance—the covert one. But so-called “open source intelligence” (OSINT) is also big business— and not just at the national/international level. New tools now mine everything from “the deep Web” to Facebook posts to tweets so that cops and corporations can see what locals are saying. Due to the sheer scale of social media posts, many tools don’t even aim at providing a complete picture. Others do.
For instance, consider BlueJay, the “Law Enforcement Twitter Crime Scanner,” which provides real-time, geo-fenced access to every single public tweet so that local police can keep tabs on #gunfire, #meth, and #protest (yes, those are real examples) in their communities. BlueJay is the product of BrightPlanet, whose tagline is “Deep Web Intelligence” and whose board is populated with people like Admiral John Poindexter of Total Information Awareness infamy.
BlueJay allows users to enter a set of Twitter accounts, keywords, and locations to scan for within 25-mile geofences (BlueJay users can create up to five such fences), then it returns all matching tweets in real-time. If the tweets come with GPS locations, they are plotted on a map. The product can also export databases of up to 100,000 matching tweets at a time.
A look at the BlueJay interface shows it to be a fairly basic tool, but one that gets its power from full access to the Twitter “firehose” of all tweets. Users who want to search the Twitterverse have three basic options: Twitter’s search API, Twitter’s streaming API, or full firehouse access to Twitter from third party providers like GNIP and DataSift. The first two are free but limited; BrightPlanet notes that even the broader streaming API returns somewhere between one and 40 percent of the relevant tweets depending on Twitter’s load at the moment. The firehose requires some serious infrastructure and a paid contract with Twitter, but it provides all relevant tweets.
BlueJay partners with firehose providers and touts this fact in its marketing copy: “BlueJay captures tweets from the entire Twitter stream, unlike all other products on the market that only get a fraction of the tweets that are being posted.” BlueJay is meant only for local monitoring of tweets, but this also keeps the price in reach of local police departments—$150 a month.
Of course, once you have this basic data, you may want to do more with it. Say a suspect is tweeting from a GPS-enabled phone client and appears to be dealing drugs. Forget bothering with the paperwork needed to track the phone through a cell phone provider. BrightPlanet also offers GeoTime, a separate data visualization tool that can take exported BlueJay data and mine it to show where and when the target travels, what he tweets about at various locations, and where his phone resides at night. (BrightPlanet describes this as using “pattern recognition to automatically detect and annotate time-space behaviors, such as meetings, gaps, connections, clusters, and motion.)
"The bad guys are out there, and they’re talking to each other online," BrightPlanet says in its sales pitch to law enforcement. “We’re intimately familiar with OSINT needs and deliverables, and we will deliver the Deep Web Intelligence that finds the bad guys and lets you get them behind bars.”