Friday, February 1, 2013

Netflix data tracking


video behind the scenes of Netflix tracking
http://whatsthebigdata.com/2012/12/24/data-science-at-netflix-with-elastic-mapreduce/





Netflix data tracking
http://www.salon.com/2013/02/01/how_netflix_is_turning_viewers_into_puppets/ 

How Netflix is turning viewers into puppets

"House of Cards" gives viewers exactly what Big Data says we want. This won't end well


I hit the pause button roughly one-third of the way through the first episode of “House of Cards,” the political drama premiering on Netflix Feb. 1. By doing so, I created what is known in the world of Big Data as an “event” — a discrete action that could be logged, recorded and analyzed. Every single day, Netflix, by far the largest provider of commercial streaming video programming in the United States, registers hundreds of millions of such events. As a consequence, the company knows more about our viewing habits than many of us realize. Netflix doesn’t know merely what we’re watching, but when, where and with what kind of device we’re watching. It keeps a record of every time we pause the action — or rewind, or fast-forward — and how many of us abandon a show entirely after watching for a few minutes.

Netflix might not know exactly why I personally hit the pause button — I was checking on my sick son, home from school with the flu — but if enough people pause or rewind or fast-forward at the same place during the same show, the data crunchers can start to make some inferences. Perhaps the action slowed down too much to hold viewer interest — bored now! — or maybe the plot became too convoluted. Or maybe that sex scene was just so hot it had to be watched again. If enough of us never end up restarting the show after taking a break, the inference could be even stronger: maybe the show just sucked.

In 2012, for the first time ever, Americans watched more movies legally delivered via the Internet than on physical formats like Blu-Ray discs or DVDs. The shift signified more than a simple switch in formats; it also marked a major difference in how much information the providers of online programming can gather about our viewing habits. Netflix is at the forefront of this sea change, a pioneer straddling the intersection where Big Data and entertainment media intersect. With “House of Cards,” we’re getting our first real glimpse at what this new world will look like.
For at least a year, Netflix has been explicit about its plans to exploit its Big Data capabilities to influence its programming choices. “House of Cards” is one of the first major test cases of this Big Data-driven creative strategy. For almost a year, Netflix executives have told us that their detailed knowledge of Netflix subscriber viewing preferences clinched their decision to license a remake of the popular and critically well regarded 1990 BBC miniseries. Netflix’s data indicated that the same subscribers who loved the original BBC production also gobbled down movies starring Kevin Spacey or directed by David Fincher. Therefore, concluded Netflix executives, a remake of the BBC drama with Spacey and Fincher attached was a no-brainer, to the point that the company committed $100 million for two 13-episode seasons.

“We know what people watch on Netflix and we’re able with a high degree of confidence to understand how big a likely audience is for a given show based on people’s viewing habits,” Netflix communications director Jonathan Friedland told Wired in November. “We want to continue to have something for everybody. But as time goes on, we get better at selecting what that something for everybody is that gets high engagement.”

The strategy has advantages that go beyond the assumption of built-in popularity. Netflix also believes it can save big on marketing costs because Netflix’s recommendation engine will do all the heavy lifting. Already, Netflix claims that 75 percent of its subscribers are influenced by what Netflix suggests to subscribers that they will like.

“We don’t have to spend millions to get people to tune into this,” Steve Swasey, Netflix’s V.P. of corporate communications, told GigaOm last March. “Through our algorithms we can determine who might be interested in Kevin Spacey or political drama and say to them, ‘You might want to watch this.’”

And maybe we will. Early reviews for “House of Cards” are promising. It will be fascinating to find out how many people gorge themselves on all 13 episodes this upcoming weekend. (Netflix data shows that’s how we like to consume our TV series now — in great gulps and marathons — so that’s how it will give them to us.) But one does end up wondering: What will the Big Data approach mean for the creative process? If Netflix perfects the job of giving us exactly what we want, when and how will we be exposed to things that are new and different, the movies and TV shows we would never imagine we might like unless given the chance? Can the auteur survive in an age when computer algorithms are the ultimate focus group? And just how many political dramas starring Kevin Spacey can we stand, anyway?

The scope of the data collected by Netflix from its 29 million streaming video subscribers is staggering. Every search you make, every positive or negative rating you give to what you just watched, is piped in along with ratings data from third-party providers like Nielsen. Location data, device data, social media references, bookmarks. Every time a viewer logs on he or she needs to be authenticated. Every movie or TV show also has its own associated licensing data. The logistics involved with handling every bit of information generated by Netflix viewers — and making sense of it — are pure geek wizardry. 

Netflix doesn’t just know that you are more likely to be watching a thriller on Saturday night than on Monday afternoon, but it also knows what you are more likely to be watching on your tablet as compared to your phone or laptop; or what people in a particular ZIP code like to watch on their tablets on a Sunday afternoon. Netflix even tracks how many people start tuning out when the credits start to roll.

Correlating the raw numbers of Kevin Spacey fans who also like David Fincher and have a fondness for British political dramas is just the beginning. Netflix knows enough about what you are watching to judge specific aspects of content as well. Last summer senior data scientist Mohammad Sabah reported at a conference that Netflix was capturing specific screen shots to analyze in-the-moment viewing habits, and the company was “looking to take into account other characteristics.”
What could those characteristics be? GigaOm’s report of the Sabah presentation speculated that “it could make a lot of sense to consider things such as volume, colors and scenery that might give valuable signals about what viewers like.”

Netflix chief content officer Ted Sarandos has said that all that data means that Netflix has a very “addressable audience.” Unlike the traditional broadcast networks or cable companies, Netflix doesn’t have to rely on shoveling content out into the wild and finding out after the fact what audiences want or don’t want. They believe they already know.

Of course, data-centric decisions don’t guarantee hit-making success. Kevin Spacey’s participation isn’t bulletproof (see “Fred Claus”) and even David Fincher can’t claim a perfect record. (“Alien 3,” anyone?) Netflix’s ambition to challenge HBO as a destination for quality original programming will require fabulous craftsmanship to go along with the Big Data filters. All the Big Data in the world can’t rule out, once and for all, the possibility of a bomb.

But that goes without saying. The interesting and potentially troubling question is how a reliance on Big Data might funnel craftsmanship in particular directions. What happens when directors approach the editing room armed with the knowledge that a certain subset of subscribers are opposed to jump cuts or get off on gruesome torture scenes or just want to see blow jobs. Is that all we’ll be offered? We’ve seen what happens when news publications specialize in just delivering online content that maximizes page views. It isn’t always the most edifying spectacle. Do we really want creative decisions about how a show looks and feels to be made according to an algorithm counting how many times we’ve bailed out of other shows?

For years Netflix has been analyzing what we watched last night to suggest movies or TV shows that we might like to watch tomorrow. Now it is using the same formula to prefabricate its own programming to fit what it thinks we will like. Isn’t the inevitable result of this that the creative impulse gets channeled into a pre-built canal?

It’s certainly possible to overstate the case here. One could argue that Netflix’s strategy is only a slightly more sophisticated version of what’s already been in place for, well, forever. We wouldn’t be seeing teenage vampires or zombies every time we turn on the TV if the money that bankrolls the content creation business hadn’t already decided that’s what we want to see. Actors who have the fortune to appear in hit movies or TV show get more parts to play. So what else is new?

But there’s a level of specificity made possible by Big Data that suggests we’re headed into new territory. “House of Cards” is just one symptom of a society-wide shift. The Obama campaign used the same kind of number crunching to target voters with more accuracy than any political campaign had ever accomplished before. Online advertisers are also gathering vast amounts of detailed information about us from our smartphones, our Facebook likes and our Google searches.

The sheer amount of data available to crunch is already phenomenal and is growing at an extraordinary rate. Last summer, at a panel discussion that included several significant players in the emerging Big Data universe, Michael Karasick, a V.P. at IBM Research, estimated that there is “a thousand exabytes of data on the planet anywhere.” An exabyte is one quintillion bytes, or 1,000 gigabytes. That’s a lot of ones and zeroes all by itself, but the mind-boggling part of the equation is that Karasick predicted that just two years from now there will be 9,000-10,000 exabytes of data on the planet.

The companies that figure out how to generate intelligence from that data will know more about us than we know ourselves, and will be able to craft techniques that push us toward where they want us to go, rather than where we would go by ourselves if left to our own devices. I’m guessing this will be good for Netflix’s bottom line, but at what point do we go from being happy subscribers, to mindless puppets?
Andrew Leonard Andrew Leonard is a staff writer at Salon. On Twitter, @koxinga21.

Wednesday, January 30, 2013

North Korean Prison Camps

My thoughts: 
 
I can only imagine that the reason for not liberating North Korea would be the instant conflict we would face with China and their allies.  N. Korea's condition has gone on for so long, it makes clear when we intervene for political gain, and that anyone touting moral obligation to justify other invasions is working spin, has another political motivation, or simply perceives limited consequences (I recall someone arrogantly claiming that we would be in and out of Iraq in three weeks, and everyone was glued to their tvs watching the initial invasion set to symphonic music).

 The same thoughts apply to Somalia - short-sightedness will ultimately lead to that region being a huge problem as the pirates grow in power and alignment with Islamic militants. As is true with the cartels in Mexico and C. America - our preoccupation with capturing the Latino vote blinds us to the concern of the crossover in cartels to the U.S. due to a lack of security and permissive policy.

http://motherboard.vice.com/blog/generations-of-chem-experiments-and-torture-the-horror-of-north-korean-prison-camps

Generations of Chem Experiments and Torture: The Horror of North Korean Prison Camps


Yesterday the Daily Mail in England broke the news that North Korean prison camps had been found on Google Earth. North Korea has long denied the existence of prison camps in its supposed utopia, but even as far back as 2004 The Guardian was reporting that the camps held an estimated 200,000 citizens. 50,000 of them, all deemed enemies of the state even though many had been actually born inside, are in North Korea’s most notorious hell, Camp 22.

And believe us, hell isn’t nearly strong enough of a word. According to reports from defectors, citizens are tossed into camps on any official whim, with most offenders being taken away for being even minutely critical of Kim Jong Il’s regime. But it’s not only the supposed offenders that are sentenced to a life of intense slave labor and malnourishment that only ends in death from exhaustion or random execution. In order to purify offenders’ blood of anti-government discontent, officials throw three generations of their family in to join them in suffering senseless torture, eating rotten food and the occasional feces and horrifying ‘scientific’ experiments involving chemical and biological weapons.

In its 2004 documentary Access to Evil, shared above, the BBC chatted with Kwon Hyuk, a former military attaché at the North Korean Embassy in Beijing. Hyuk, who changed his name, was also the former chief of management at Camp 22. He defected upon seeing the light, but his descriptions of experiments he watched over are simply horrifying. Entire families, along with random single prisoners, were (and possibly still are) placed in giant glass boxes. While pumping in poisonous gasses, scientists stood on top of the box, recording the reactions of the dying families below.
“I witnessed a whole family being tested on suffocating gas and dying in the gas chamber,” he said in the BBC doc. “The parents, son and a daughter. The parents were vomiting and dying, but till the very last moment they tried to save kids by doing mouth-to-mouth breathing.”

Perhaps even more frightening is Hyuk’s admission that he felt absolutely no remorse at the time for any of the prisoners. According to him, they were simply enemies of the state, and deserved to die.
The BBC also talks to Sun Ok-Lee, who was one of the few prisoners to be released from the camps after it was deemed she was sufficiently reformed. While talking of the torture she endured in her seven years there, she describes an experiment she took part in.

“An officer ordered me to select 50 healthy female prisoners,” she said. “One of the guards handed me a basket full of soaked cabbage, told me not to eat it but to give it to the 50 women. I gave them out and heard a scream from those who had eaten them. They were all screaming and vomiting blood. All who ate the cabbage leaves started violently vomiting blood and screaming with pain. It was hell. In less than 20 minutes they were quite dead.”

Most of the info on the camps from the interviews in Access to Evil were corroborated by a 2008 Washington Post story on Shin Dong-hyuk, the only person known to have successfully escaped from a North Korean prison camp. Dong-hyuk was actually born in his camp. At one point he was tortured and locked in an underground room for seven months because his mother and brother were supposedly hatching an escape plot. He later saw them both executed.

The importance of the Daily Mail‘s report is that it lends a whole lot of weight to past prison camp reports. North Korea has always denied the existence of the camps, and with independent verification impossible in a country that so tightly controls foreign press, there’s always that shadow of a doubt (At least in the public eye. It’s hard to imagine that the imaging resources of Google Earth outpaced government agencies in discovering the camps). Hell, 2008’s Vice Guide to North Korea provided some of the most in depth information on the country many people had ever seen and it required journalists to sneak into the country and, as VICE is apt to encourage, leverage hard partying to gain exclusive access to information. What’s paramount, as grinding diplomacy plods back and forth in the region, is reminding the public of the existence of these horrors is key in pushing leaders to demand concessionary change.

Monday, January 28, 2013

Cyber War On the US Has Begun

http://www.infoworld.com/d/security/unseen-all-out-cyber-war-the-us-has-begun-211438
By

Security pros and government officials warn of a possible cyber 9/11 involving banks, utilities, other companies, or the Internet

The Senate’s immigration plan

http://www.washingtonpost.com/blogs/wonkblog/wp/2013/01/28/the-5-most-important-sentences-in-the-senates-immigration-plan/

The 5 most important sentences in the Senate’s immigration plan

Posted by Ezra Klein

Have you read the Senate’s bipartisan immigration-reform framework yet? At four pages, it’s a quick and clear read. But because it’s only four pages, quite a bit is left unsaid, or undefined. So here’s what you should keep a particularly close eye on:
Erik S. Lesser/AP
(Erik S. Lesser/Associated Press)

1) “Contingent upon our success in securing our borders and addressing visa overstays.”
The key word in the framework is “contingent.” Everything is contingent upon the securing of our borders. But how will we known when our borders are secure? “Our legislation will create a commission comprised of governors, attorneys general, and community leaders living along the Southwest border to monitor the progress of securing our border and to make a recommendation regarding when the bill’s security measures outlined in the legislation are completed.”

As Greg Sargent writes, “The fate of immigration reform, then, largely rests on what this commission looks like, who is on it, and what metric it uses to decide when the border is secure.”

2) “Our legislation also recognizes that the circumstances and the conduct of people without lawful status are not the same, and cannot be addressed identically.”

There’s a lot you have to do to qualify for the path to citizenship in this bill. “Go to the back of the line of prospective immigrants, pass an additional background check, pay taxes, learn English and civics, demonstrate a history of work in the United States, and current employment, among other requirements.”

But that’s not true for everyone. The framework makes clear that the law won’t treat all undocumented immigrants identically. The obvious exception is children, and sure enough, they get a special mention: “individuals who entered the United States as minor children did not knowingly choose to violate any immigration laws. Consequently, under our proposal these individuals will not face the same requirements as other individuals in order to earn a path to citizenship.”

The more unexpected exemption is agricultural workers. “Individuals who have been working without legal status in the United States agricultural industry have been performing very important and difficult work to maintain America’s food supply while earning subsistence wages. Due to the utmost importance in our nation maintaining the safety of its food supply, agricultural workers who commit to the long term stability of our nation’s agricultural industries will be treated differently than the rest of the undocumented population.”

3) “Our new immigration system must be more focused on recognizing the important characteristics which will help build the American economy and strengthen American families.”

Perhaps the key architectural question in building a new immigration system (as opposed to figuring out what to do with the failures of the last immigration system) is deciding whether we’ll focus our visas on drawing needed skills (for instance, by favoring immigrants with advanced degrees or who work in sectors, like agriculture, where we need more labor), emphasizing family unification or something else.

This paper mentions both family reunification and skills. And to some degree, that’s fine: We don’t need to choose simply one or the other. But since we’re not moving to an open-border policy, the fact that we’re offering a limited number of visas will mean that we have to choose how many of them go toward  reuniting families and how many go toward filling our perceived economic needs. Those ratios are left undefined in this framework, but they’ll be one of the most hotly contested elements of the actual law.

4) “Our immigration proposal will award a green card to immigrants who have received a PhD or Master’s degree in science, technology, engineering  or math from an American university.”
Remember when Mitt Romney said, “If the [immigrant] student does so well that they get an advanced degree, I’d staple the green card to their diploma”? This law basically does that.

5) “Our proposal will provide businesses with the ability to hire lower-skilled workers in a timely manner when Americans are unavailable or unwilling to fill those jobs.”

This refers to some kind of guest worker program, though the details are sparse and the language is vague. The section mostly focuses on the hoops employers would have to jump through, including demonstrating “that they were unsuccessful in recruiting an American to fill an open position and the hiring of an immigrant will not displace American workers.” But guest-worker programs are always controversial, and how big this program is, how difficult the qualifications are, who qualifies, etc., are all left unresolved in this paper.

As just one example, consider the complexities in implementing this sentence, which is all the paper says about how to move out of the guest worker program and into green-card status: “Permit workers who have succeeded in the workplace and contributed to their communities over many years to earn green cards.”

Wednesday, January 23, 2013

NYC drinks limit raising questions of racial fairness


 If a question is raised about fairness to low income areas or businesses owned or patronized by racial minorities, why not eliminate that factor entirely and focus restricting what people can purchase with EBT cards?


Beverage attorney: NYC drinks limit bad for public



NEW YORK (AP) -- New York City's limit on the size of sugary drinks is an "extraordinary infringement" on consumer choice, a lawyer for the American Beverage Association and other critics said in court on Wednesday.

"New Yorkers do not want to be told what to drink," attorney James Brandt told Manhattan state Supreme Court Justice Milton Tingling.

Opponents also are raising questions of racial fairness alongside other complaints as the novel restriction faces a court test.

The NAACP's New York state branch and the Hispanic Federation have joined beverage makers and sellers in trying to stop the rule from taking effect March 12. Critics are attacking what they call an inconsistent and undemocratic regulation, while city officials and health experts defend it as a pioneering and proper move to fight obesity.

The issue is complex for the minority advocates, especially given that obesity rates are higher than average among blacks and Hispanics, according to the federal Centers for Disease Control and Prevention. The groups say in court papers they're concerned about the discrepancy, but the soda rule will unduly harm minority businesses and "freedom of choice in low-income communities."

The latest in a line of healthy-eating initiatives during Mayor Michael Bloomberg's administration, the beverage rule bars restaurants and many other eateries from selling high-sugar drinks in cups or containers bigger than 16 ounces. Violations could bring $200 fines; the city doesn't plan to start imposing those until June.

The city Board of Health approved the measure in September. Officials cited the city's rising obesity rate - about 24 percent of adults, up from 18 percent in 2002 - and pointed to studies linking sugary drinks to weight gain. Care for obesity-related illnesses costs more than $4.7 billion a year citywide, with government programs paying about 60 percent of that, according to city Health Commissioner Dr. Thomas Farley.

"It would be irresponsible for (the health board) not to act in the face of an epidemic of this proportion," the city says in court papers. The National Association of Local Boards of Health and several public health scholars have backed the city's position in filings of their own.

Opponents portray the regulation as government nagging that turns sugary drinks into a scapegoat when many factors are at play in the nation's growing girth.

The American Beverage Association and other groups, including movie theater owners and Korean grocers, sued. They argue that the first-of-its-kind restriction should have gone before the elected City Council instead of being approved by the Bloomberg-appointed health board.

Five City Council members echo that view in a court filing, saying the Council is "the proper forum for balancing the city's myriad interests in matters of public health." The Bloomberg administration counters that the health board, made up of doctors and other health professionals, has the "specialized expertise" needed to make the call on limiting cola sizes.

The lawsuit also argues the rule is too narrow to be fair. Alcohol, unsweetened juice and milk-based drinks are excluded, as are supermarkets and many convenience stores - including 7-Eleven, home of the Big Gulp - that aren't subject to city health regulations.

The NAACP and the Hispanic Federation, a network of 100 northeastern groups, say minority-owned delis and corner stores will end up at a disadvantage compared to grocery chains.

"This sweeping regulation will no doubt burden and disproportionally impact minority-owned businesses at a time when these businesses can least afford it," they said in court papers. They say the city should focus instead on increasing physical education in schools.

During Bloomberg's 11-year tenure, the city also has made chain restaurants post calorie counts on their menus and barred artificial trans fats in french fries and other restaurant food.

In general, state and local governments have considerable authority to enact laws intended to protect people's health and safety, but it remains to be seen how a court will view a portion-size restriction, said Neal Fortin, director, Institute for Food Laws and Regulations at Michigan State University.
---
Follow Jennifer Peltz at http://twitter.com/jennpeltz

Hobbyist builds working replica of Iron Man's laser gauntlet


More amazing designs here from Patrick Priebe:
http://www.laser-gadgets.com/laser_gloves.php



Gallery from the article
 http://www.gizmag.com/iron-man-laser-gauntlet/25901/pictures

Article:

http://www.gizmag.com/iron-man-laser-gauntlet/25901/

Given that most real-life superheroes don’t have the budget of Tony Stark or Bruce Wayne, you would assume that their gadgetry wouldn’t be quite on par with what we’re used to seeing in the movies. German cyber weapons hobbyist Patrick Priebe recently dropped us a line, however, to tell us about his latest homebuilt creation – a working laser gauntlet, just like the one made famous by a certain Iron Man.

The aluminum-bodied device opens up for the user to insert their forearm, then gently clamps onto it using a spring/lever system. An LED display on the side of the gauntlet lights up once it’s locked in place, letting the user know that it’s ready to go ... and also to look cool, of course.

When the would-be Iron Man wants to zap something, they do so via button controls on the palm-mounted control module. This initially activates a servo, which causes the laser rig to rise out of the top of the gauntlet. Two red lasers on that rig are then used for aiming at the target, after which a thicker blue laser (mounted between them) is selected to deliver the coup de grâce. It may not be a strong enough beam to disable a car or anything, but as you can see in the demo video at the bottom of the page, you wouldn’t want to be a balloon around this thing.

Although Iron Man’s movie gauntlet utilized a red "destructive" beam, Priebe told us that he went with a blue laser because it’s more powerful and more visible. Additionally, a second blue laser is located in his gauntlet’s palm control unit. All four lasers are powered by one 3.7-volt lithium-ion battery, with two smaller batteries powering the LEDs and the servo.

According to Patrick, it took him 120 hours to build the gauntlet – and he wasn’t working from plans of any kind. As with his past creations (such as a coilgun, flame-throwing glove, and rotary blade-shooting crossbow), he’s not about to tell people how to build one of their own. If you contact him via his website, however, he might be willing to make you one ... for the right price.