Facebook's Content Review

Not too surprisingly, Facebook outsources its content review to a company called oDesk, which uses workers in developing nations.  Gawker has the story in more detail, including copies of Facebook's guidelines.

Lydia Cacho Takes on Facebook

Last month, the Daily Beast ran an article about Facebook taking heat from activist Lydia Cacho about the sexual predators.  It's not too surprising that their perspectives differed.

Perhaps we can shed some light on the issue.  First, let's start with something that both the Facebook safety team and Cacho would probably agree on:  most people strongly dislike child pornography.  However, many of the pedophiles who collect the stuff are apparently somewhat compulsive about it.  When they get busted, they generally don't just have a handful of images; it's more like hundreds or even thousands.  

The internet makes it a lot easier for pedophiles to exchange material with like-minded people.  Peer-to-peer software has often been used for this, but it's not too hard for law enforcement to automatically scan for known child pornography files on P2P systems and pick off the idiot who's sharing it.  Social networks allow pedophiles to look for validation and acceptance as well as trading partners, so it's not too surprising that they've shown up on Facebook, Grou.ps, Grouply, MySpace, Google+, etc.

Facebook is currently the biggest player in the social network arena, with over 500 million accounts.  They've made an effort to curb predators by scanning for known child pornography and certain kinds of anomalous behavior.  But it's very hard to detect some things automatically, and it's impossible for them to do it manually.  Therein lies the rub.

Joe Sullivan, Facebook's security chief, used an example of an account that sent all its friend requests to teenage girls.  However, predators may also be looking to connect with other pedophiles.  He also pointed out that Facebook is using PhotoDNA to scan photos.  PhotoDNA is basically an improved approach to detecting known child pornography.  Earlier this year, a man in the UK took photos of himself sexually abusing his 2-year-old daughter and uploaded them to Facebook so that other like-minded users could view them.  PhotoDNA would not have been able to flag this because the image would not have been in its database.  So it's commendable of Facebook to make the effort to use this technology, but it's still no silver bullet.

Because of the sheer volume, a lot of the Facebook account shutdowns are done automatically, with no human intervention.  In the example Sullivan provided, the system might notice that a lot of friend requests are being sent to teenage girls and disable the account.  However, if an account that tripped off some flags and got automatically shut down was also posting child pornography that PhotoDNA did not detect, no human would see it and it would consequently not get reported to law enforcement.  It's possible that the failure to report some of this material is related to this, rather than willfully ignoring the law.

If Facebook continues to work on combating predators, it may encourage many of them to relocate to a different social network.  But that's probably about as good as it's going to get.

PhotoDNA

According to numerous articles and press releases, Facebook will be using a new software package called PhotoDNA developed by Microsoft and donated to NCMEC.  It uses a technique called "robust hashing" and is based on work by Dartmouth's Hany Farid.

It's easy to demand that something be done about child pornography, but much harder to actually curb it.  Computer vision is useful for simple things (like reading license plate numbers), but it has its limitations.  Most people have experienced the "CAPTCHA" technology to block spam where you have to read a series of letters or do something similar.  Why does that work?  Because humans can recognize things in images that computers can't.

However, it is possible for a computer to tell whether an image is identical to known child pornography.  Law enforcement and ISPs currently use a type of hashing known as SHA-1.  ISPs would calculate the SHA-1 hashes of images uploaded to their servers, compare them to hash values of known child pornography, and then take action if they detected a match.  An automated approach like this is very important for big players like Facebook that may have millions of images uploaded every day.  Having humans review all content isn't feasible.

The problem with the SHA-1 hashing was that it would only work if one image was exactly identical to another.  So let's say someone took a known CP image and slightly cropped or resized it.  The automated image recognition breaks because the files are no longer exactly the same.  This is what PhotoDNA addresses.  Note that even PhotoDNA only works with known CP images.  So if a predator creates his own CP and uploads it, the system won't recognize it.  (This happened with John Huitema, who recently pleaded guilty to victimizing a 2-year-old girl and producing child pornography.)

The information about how PhotoDNA works is pretty limited, since it's mostly in the form of Microsoft press releases.  So there are still a number of unanswered questions.  It sounds like the image is converted into black and white, resized to a standard size, broken up into small blocks, and the digital signature is calculated for each block.  So this is supposed to detect images that have been altered, and Microsoft said that their testing had yielded some promising results.

It does sound like PhotoDNA will be more robust than SHA-1 hashing, but there's not much information available on its limitations or future directions.  It's also not possible for regular people to download the software and play with it.  Some image manipulation involves discarding a lot of information.  Would PhotoDNA be able to detect a preview-sized image of known child pornography that someone was advertising on another site?  The "digital fingerprint" seems to rely a lot on edge detection and intensity.  What happens if someone alters a photo in a way that changes the edge information, e.g., by adding another object or lettering?

Some of the Facebook problem children have also shared CP videos.  It does not sound like PhotoDNA is currently being applied to that, and I'd be curious what the potential was in this area.   Some video codecs contain I-frames, which could probably be analyzed this way.

Ironically, Windows Live has apparently had some major problems with child pornography.  Has Microsoft tried applying PhotoDNA to its own network?  If not, why not?  If so, why doesn't it work better?

Recent Arrests

Recently, there have been more arrests in the U.S..  Arthur Barnett, Tong Yang, and Stephen Schmid were charged after Facebook reported them to NCMEC for posting child pornography.  This is encouraging; it seems like there's been some real improvement.  Congratulations to Facebook, NCMEC, and the law enforcement agencies involved.

Ars Technica

Facebook shut down Ars Technica's page, accusing them of copyright infringement.  Given that Ars Technica was apparently just posting their own material to their page, it's not entirely clear where the alleged infringement would have occurred.  Facebook hasn't given them any hints either.  The story is up at  http://arstechnica.com/business/news/2011/04/facebook-shoots-first-ignores-questions-later-account-lock-out-attack-works.ars

Update (6:34 PM CST):  The Ars Technica page was restored.  Facebook is apparently working on an official statement.

Two Men Kissing Update

Via Newser, Facebook has apparently apologized for removing the photo of two men kissing.  In fairness to Facebook, the event was apparently removed by the organizer in response to troll harassment and not by Facebook itself (as the blog post originally stated).

Two Men Kissing

Some users organized a "Kiss-In" protest on Facebook in response to an incident where two gay men were ejected from a pub in the UK.  One man posted a photo of two men kissing to the "Dangerous Minds" Facebook page in the context of the event.  Facebook removed the photo, sent the user a warning, and deleted the protest event.  Below is a screenshot from their blog that includes the photo.

Meanwhile, let's take a look at what some currently active users on Facebook have been up to.  Sarah Dee is interested in finding some other groups where people post images and videos.

What kind of images and videos?  Well, Sarah Dee lists her interests as "Lolita" and "P T H C."  For the blissfully unaware, "PTHC" stands for "pre-teen hardcore."  We can reasonably assume that Sarah and friends are interested in stuff that's a lot more explicit than kissing, and it doesn't involve adults.

Sarah Dee's friend "Children Side" is also openly trolling for child pornography.

Emily Emmerson uses the "Questions" application to ask "Would anyone like to trade pics or vids?"

Zac Cartoon is also looking for like-minded friends and is not too subtle.  He posts, "plz all pedo add me."

The two fully clothed men in the first picture hardly seem sexually explicit.  If the Facebook safety team would like to see something that was, they might want to take a look at some of the closed and private groups that some of the users we saw were talking about.

Reporting Defaults

Most of the Facebook users interested in child pornography take it somewhere else or use private groups.  However, one will occasionally post some horrendous stuff to an open wall where anyone can see it.  Recently, I came across this and reported it to Facebook.  Here's what the reporting interface looks like.    We select the "nudity or pornography" category.

Here's what's a little worrisome.  Bear in mind that we're trying to report child pornography.  So why is the default option set to just block the user and not report it to Facebook?

Sheva and Joe

This summary is not available. Please click here to view the post.

Police in Australia Criticize Facebook

According to a TNW story at http://thenextweb.com/facebook/2011/03/24/australia-police-claims-facebook-hinders-child-abuse-probe/ , Australian police have expressed frustration at the mutual assistance treaty and Facebook's response.

Australian Federal Police’s high-tech collection and capability manager, Grant Edwards, expressed that the mutual legal assistance treaty with the US is outdated and not designed to allow for a rapid response to online crime. And that Facebook, the world’s biggest social network, hasn’t been helpful with the push.

A revised treaty, supported by the AFP, is necessary to speed up the actioning of the police’s right to obtain information about online crime using a warrant on Facebook, which is head-quartered in the United States. Currently, the AFP is still required to undergo a long, tedious legal process in order to use US law enforcement to force Facebook to respond.

The Australian government has made efforts to amend the assistance treaty in order to speed up the process of lawful data acquisitions. However, the changes would need to be ratified by the US to take effect. According to Edwards, Facebook hasn’t been as helpful as it could because the company is “brand-protecting.” Furthermore, he labeled the social networking site’s action to remove pedophile profiles that were needed as evidence by the AFP as "ignorance on Facebook’s behalf."

Facebook has an automated system to detect and disable problem accounts.  However, this system generally can't detect when a user should also be reported to law enforcement.  Once an account is shut down, Facebook will retain the information for up to 90 days, and then it's gone.  Automatically detecting child pornography is a hard problem.  The most promising approach has been comparing images to hashes of known child pornography files, but that only works if there's a match.

The situation may also be fraught in terms of politics and public relations.  The ground truth seems to be that there's a lot of child pornography floating around and nobody's really figured out a surefire way to detect it.  Even if it is detected and reported to law enforcement, some countries are more responsive than others.  But a social network can't go on record saying that child pornography is a hard and frustrating problem; it's more palatable to say that their sophisticated technology is keeping things under control.  As an obscure blogger, I can comment about seeing a lot of users from certain countries who seem to be interested in child pornography.  If a Facebook spokesman said that, people would probably accuse them of being bigots and they'd be forced to apologize.

Reflections and Expectations

In the few months since this blog started, a lot has changed.  In November when it started, there were a number of users on Facebook blatantly advertising their interest in child pornography.  Some would even post links and invitations to trade on open walls.  Facebook has since gotten more aggressive about shutting those accounts down, though it may never be possible to completely block predatory users.  A number of concerned citizens have gotten very upset about the situation, and some have even set up a website and sent postcards to Facebook.

There may be some conflict between expectations and reality.  When I started this blog, most of my understanding of child pornography had come from news stories about people getting arrested.  There weren't too many of them in my community, and I was inclined to assume that very few people were interested in child pornography and the ones who did got quickly arrested.  So it was hard to understand why some of the people I saw weren't in jail.

Since then, I've learned that the sheer volume of child pornography on the Internet is staggering, and none of the major social networks are completely free of predators.  There are a number of dedicated law enforcement professionals working hard to combat the problem, but the situation has been compared to drinking out of a fire hose.

Why can't Facebook and other social networks completely block pedophiles and child pornography from their sites?  It turns out to be a much harder technical problem than many activists realize.  It is often impossible for a system to recognize that an image is child pornography and when there are millions of accounts, it's not feasible for a human to manually review all images and videos that users post to a site.  Child pornography traders generally try not to call attention to themselves and are less likely to get reported by other users than spammers or a malicious trolls.  Even when someone who posts child pornography is detected and reported to law enforcement, their country may or may not take action.

A number of concerned citizens are currently demanding that Facebook eliminate child pornography from its site.  It's easy to understand why people are upset and angry about child exploitation, but harder to accept that there's only so much Facebook or any other social network can do about it.  The best case scenario may be that some people get arrested and most of the child pornography traders migrate elsewhere.  But no matter how proactive the social networks are, child pornography is going to continue to be a hard problem.

Missouri Arrest

According to the Columbia Tribune, John Kitch was arrested last week for posting child pornography to Facebook.  Congratulations to Facebook, NCMEC, and the Boone County Cyber Crimes Task Force.

John W. Kitch of 5809 S. Bethel Church Road was arrested yesterday and was being held in the Boone County Jail with bond set at $100,000.

The Boone County Sheriff Department’s Cyber Crimes Task Force began investigating the incident in December after Facebook employees discovered the child pornography and reported it to the National Center for Missing and Exploited Children. 

Back Again

Marcos Teia may create new Facebook accounts more frequently than he changes his underwear.  It's hard to tell.  But he inevitably pops up every day or so with yet another name or user photo.  Within a few hours, Facebook has usually shut down not only that account but all of his friends too.  I'm betting that the one below plus all of his friends will get nuked by noon tomorrow.  Mr. Teia has also decided to grace grou.ps with his presence again, along with some reportable photos of young girls.  Hopefully, Facebook security won't get too jealous over this since they'll almost certainly see him again.

BTW, it might be worth checking out his groups before booting him.   He's apparently found his way into at least one that may be sharing something other than Bible verses and holy cards.

Update: They didn't even last until 9 AM Pacific Time.  The group that he was in is gone too.

Facebook and the Daily Mail

There's a fascinatingly convoluted blog post today entitled "Time for a referee between Facebook and the Daily Mail" by Dan Sabbagh.  This was apparently related to a possible lawsuit about a Daily Mail headline that said "How many victims of Facebook sex gang?"  When pressed, the Daily Mail changed its wording of the headline, but refused to apologize.

In early 2010, the Daily Mail claimed that a man posing as a 14-year-old girl who had created a profile on Facebook had been approached "within seconds" by older men who wanted her to perform a "sex act" in from of them.  Turned out that the author had actually created the profile on a different social network that had nothing to do with Facebook.  Oops.

In a nutshell, I'm not convinced that there's any need for a referee here, but some basic fact-checking from the Daily Mail might be a good start.

Examiner Article

We were recently mentioned in an Examiner article by Raymond Bechard, who has written extensively about human trafficking.

To be fair to Facebook, dealing with abusive accounts is a hard problem and they've been making some effort.  They've got a system in place that seems to be automatically nuking a number of profiles.  However, the bad guys just come right back with new accounts, and I'd doubt that there's any mechanism in the "auto-nuke" system to report predatory users to law enforcement.  If not, that could mean that the only consequences some of the people are experiencing for soliciting and posting child pornography are having a Facebook account shut down.

Here's an interesting article from last year about Facebook not notifying Australian LE about child pornography:
http://www.smh.com.au/technology/facebook-failed-to-tell-police-about-paedophile-porn-ring-20100826-13ual.html
It says that someone had sent Facebook 10 messages telling them about the CP rings and nothing had happened.  That was likely more about the inability to process user messages than deliberate refusal to do anything about child pornography.

Facebook's reporting mechanisms are limited.  For example, imagine you see a profile of of a man hinting at sexually abusing his daughter.  How do you go about reporting that to Facebook?  Note that there is no "child exploitation" option anywhere.  If the profile picture contains actual nudity or pornography, that's reportable, but if they're just using a photo of a child to signal their interests, no dice.  "Inappropriate profile information" is limited to hate or violence, and there's nothing for profiles that refer to pedophilia or child porn.  There's also no free text field that allows you to provide additional information.

Realistically, Facebook has millions of users.  If you assume that 1% of accounts are abusive in some way but there are 100 million accounts ... well, that's potentially 1 million abusive accounts.  That's why Facebook wants to automate as much as it can -- they have to.  However, I would like to see an improved reporting system that allowed users to specifically flag child exploitation and other things that are problems but don't fit neatly into their existing system.  For example, how would you go about reporting a profile or page that seemed to be advertising an escort service (human trafficking)?

Traffic

While watching my blog traffic a few weeks ago, I came across a referral link to my blog from this site:

So I watched the traffic some more.  There'd be a hit from the same ISP associated with the Facebook intern site, and some of the profiles featured in the blog would be taken down shortly afterward.

This is fine - certainly better than the numerous hits I get from users searching for "pthc" and related terms.  It looks like Google/Blogger may also be paying occasional visits, which is reasonable.  So greetings, Facebook intern and Google content reviewers.

Content review is potentially expensive, because there's no good automated way to do it and human labor costs money.  Relying on users to report each other is slightly more effective, but has some big problems too.  Users with unpopular opinions can get mass-reported, but accounts using private groups to exchange horrendous stuff fly under the crowd's radar.

A better reporting system could help.  For example, the available choices to report a user are "Inappropriate profile photo", "Fake profile", "Inappropriate profile info", "Inappropriate wall post", and "Unwanted contact."  What about something related to child exploitation that would hopefully get higher priority than a Facebook account someone had set up for their dog?  Likewise, there's a general "nudity/pornography" category to use in reporting photos, but nothing specific for child pornography.

One simple trait that we've seen with some of the problematic users we've tracked is that they frequently get shut down and come right back with new accounts, sometimes on the same day.  Slashdot and some other sites use the notion of "karma," that rewards users for positive (or acceptable) behavior over a period of time.  Facebook could require that users accounts wait a certain amount of time to access certain features.  For example, what if it took 1 week with no warnings or profile name changes to be allowed to join closed groups, and 2 weeks to join private ones?

Facebook's response

For some reason, a group of people got upset about the user who'd been known as "Todd Lerfuk" and "Peetee Achcee."

They didn't care much for Mr. Hussyfan either.  So they made some strongly-worded complaints, and Facebook issued a warning -- to them.

Here's one complaint about Mr. Hussyfan, originally posted to http://www.facebook.com/permalink.php?story_fbid=161257963912264&id=152175601493274 and retrieved from Google's cache, that was removed.