Republished on February 28 with Google and user response to this photo scanning furor, and more information on the system update warnings.
You may recall Apple’s un-Apple-like moment a few weeks ago, when users discovered their photos were being scanned by Apple Intelligence to match landmarks. Users had not been told, and it caused a furor with security experts. Google is now going through something of the same. And again, it’s not the technology, it’s the secrecy.
Apple’s Enhanced Visual Search sends parts of photos to the cloud to match against a global index of points of interest. It’s very privacy-preserving, but as crypto expert Matthew Green complained, “it’s very frustrating when you learn about a service two days before New Years and you find that it’s already been enabled on your phone.”
Google’s awkward moment relates to its SafetyCore, an Android system update that enables on-device image scanning that could do all kinds of things, but is currently focused on blurring or flagging sensitive content. It’s seemingly even more private than Apple’s Enhanced Visual Search, given that it’s all on-device. So we’re told.
But when a technology is installed and enabled on our phones without warning, the after-the-fact assurances that it’s all fine tend to be met with more skepticism than would be the case if it was done more openly. That’s the same issue as Apple’s.
The X post that kicked off this SafetyCore furor warned “Google had secretly installed this app on various android devices without users permission. It can reportedly scan through your photo gallery and occupies 2gb of space.”
I have covered SafetyCore before, pointing out that its use to secure Google Messages would be a welcome addition to Gmail, shifting security scanning from Google’s servers to a user’s phone. But that doesn’t change the lack of openness point.
GrapheneOS — an Android security developer — provides some comfort, that SafetyCore “doesn’t provide client-side scanning used to report things to Google or anyone else. It provides on-device machine learning models usable by applications to classify content as being spam, scams, malware, etc. This allows apps to check content locally without sharing it with a service and mark it with warnings for users.”
But GrapheneOS also points out that “it’s unfortunate that it’s not open source and released as part of the Android Open Source Project and the models also aren’t open let alone open source… We’d have no problem with having local neural network features for users, but they’d have to be open source.” Which gets to transparency again.
Google says that SafetyCore “provides on-device infrastructure for securely and privately performing classification to help users detect unwanted content. Users control SafetyCore, and SafetyCore only classifies specific content when an app requests it through an optionally enabled feature.”
And once users know it’s there, that’s all true.
Per ZDNet, the issue is that the “Google never told users this service was being installed on their phones. If you have a new Android device or one with software updated since October, you almost certainly have SafetyCore on your phone.” As with Apple, “one of SafetyCore’s most controversial aspects is that it installs silently on devices running Android 9 and later without explicit user consent. This step has raised concerns among users regarding privacy and control over their devices.”
Google emphasizes that while SafetyCore brings the architecture to scan your photos, the scanning itself is done separately, for example with the Sensitive Content warnings rolling out this year, and that it’s all done on device.
As for the secrecy, Google told me “Google System services automatically updates your device with security, bug fixes, and new features. Some updates are delivered via system services in separate Android packages. This maintains privacy, security and data isolation following the principle of least privilege because permissions are not shared with other functionality. As part of Google’s continuous investment in transparency of its products, we added binary transparency to these Google system APKs.”
SafetyCore was covered in November when it was released, but it hasn’t generated any real media attention until now. Google did provide an overview of its development capabilities at the time, and it separately promoted the upcoming sensitive content warnings coming to Google Messages, similar to Apple’s on-device content safety.
But the issue this has highlighted is different. There’s a user nervousness around what all the clever new tech is doing on our phones, and with Google maybe more than most, the delineation between on and off device is often lost. There’s a trust issue that comes from the publicity around tracking and data harvesting that won’t quickly fade.
Google stresses that users remain in control, that they can disable or uninstall SafetyCore and they don’t need to enable the on-device scanning when it comes. I suspect there needs to be some more PR around the privacy and the benefits of the new functionality, or trigger-happy users will read the coverage and switch it off.
Per one tech forum this week: “Google has quietly installed an app on all Android devices called ‘Android System SafetyCore’. It claims to be a ‘security’ application, but whilst running in the background, it collects call logs, contacts, location, your microphone, and much more making this application ‘spyware’ and a HUGE privacy concern. It is strongly advised to uninstall this program if you can. To do this, navigate to ‘Settings’ > ‘Apps’, then delete the application.”
If you “don’t trust Google,” because as ZDNet points out, “just because SafetyCore doesn’t phone home doesn’t mean it can’t call on another Google service to tell Google’s servers that you’ve been sending or taking ‘sensitive’ pictures,” then you can stop it. You can find the option to uninstall or disable the service by tapping on ‘SafetyCore’ under ‘System Apps’ in the main ‘Apps’ settings menu on your phone.
Lessons learnt for both Apple and Google in recent weeks then. If you want to turn our phones into AI-fueled machines, then let us know what you’re doing before you do it, and give us the opportunity to say yes or no. Otherwise it fuels fear of the unknown. And if AI is to bed down on our smartphones with access to all our apps and data, then it needs to establish high trust bars and stick to them rigidly.
Meanwhile, the furor around SafetyCore continues to build, with it clearly struggling to shake off the spyware implication of a system app with this underlying function being installed without notice or a specific opt-in/opt-out assurance.
“I did not consent to this app being installed, nor was I notified,” complained one Redditor. “And when I found out about this through another source, I couldn’t find this app by using Google Play’s search function. I had to use a link someone else provided to even locate it. (Some people said they found and uninstalled it through their security settings.) Absolutely obscene. Apparently it’s been added to older devices as well – even crashing some of them.”
While a review on Google’s own Play Store picked up the same theme, warning: “For anyone wondering: This app is designed to look at EVERY image received by your device, to look for obscene images and blur them. Google claims it won’t ever send the images off your phone to look, but we all know that’s not true. To Google: I only learned that you installed this cause one of my security apps warned me. Don’t force stuff on us without so much as a notification, that’s straight up malware. Especially when it’s something that’s designed to impeded MY use of MY phone.”
But there were some comments that balanced the debate, focusing on the privacy preserving aspects of SafetyCore that Google is keen to emphasize.
“Lots of disinfo around Google ‘secretly scanning your messages and sending it to the cloud’,” commented another Redditor. “Total nonsense. I don’t understand how people get away with saying be like this and their followers just eat it all up. Analogy from the article about what SafetyCore actually does: ‘So this bouncer uses AI to spot shady stuff like spam, scams, malware, and even those NSFW pics (yikes!) in your messages and apps. The best part? It does all this without snitching to Google or anyone else. Think of it like a super-smart security guard who can spot trouble without calling the cops. By not snitching to Google or anyone else or calling the cops, it’s not sending your information to anyone’.”
But as ever, such balanced views when it comes to any form of scanning will always be in the minority. As one Redditor responded to the above comment: “I think these kinds of apps are built on trust and right now trust in Google is not that stellar and when these kinds of apps install themselves like malware, people will not trust it outright even if they have good intentions. It’s like installing Adobe Reader and it also downloads McAfee antivirus cause, you know it’s gonna protect you from so-called ‘viruses’. Or an overprotective helicopter parent that scans your text as you’re asleep and blocks the number of friends they deem not a good influence on you.”
Google does warn that such updates will happen in the background. “Google System updates give you new and useful features that make your Android devices more secure and reliable. It includes updates to the Android operating system and key system services provided by Google, Google Play Store, and Google Play services. Google System updates are available for all Google-certified Android devices.”
It also says that “by default, Google System services automatically updates your device with security, bug fixes, and new features. Some updates are delivered via system services in separate Android packages. This maintains privacy, security and data isolation following the principle of least privilege because permissions are not shared with other functionality. These separate Android packages also benefit from enhanced transparency.”
I think it’s safe to assume that most users are unaware of this or assume such updates are benign. Even if, as Google explains, “during device setup, users acknowledge that: “this device may also automatically install updates and apps from Google, your carrier, and your device’s manufacturer.”
Suffice to say, for Google (and Apple and others), the small print is no longer enough. Enhanced Visual Search and now SafetyCore tells us that next time this needs to be clearly spelled out from the beginning, we don’t want this to become the norm.