Apple has made its App Store moderation rules explicit about nonconsensual intimate imagery, including images generated with AI. The October 16 update says affected apps must proactively prevent and remove this content, and developers risk removal when safeguards are inadequate or violations go unaddressed after notice.

Apple says its existing pornography prohibition already covered apps that generate sexually explicit images of real people without their consent. The change spells out responsibilities within the content moderation rules and clarifies consequences for developers.

The big change

  • What changed: Apple explicitly places AI content creation apps within its moderation requirements. Developers must account for generated content as well as material users upload or share.
  • Why it matters: The rules connect safeguards and responses to abuse with the developer's continued access to the App Store. For teams building image tools, moderation is part of the product's eligibility for distribution.
  • What to watch: Apple leaves the choice of safeguards unspecified. Its decisions about remediation and removal will show which protections it accepts when assessing an app's compliance.

Moderation includes content an app generates

Guideline 1.2 now names AI content creation apps alongside social media and other platforms where users contribute content. It identifies nonconsensual intimate imagery among the risks those services present.

The guideline requires filtering before objectionable material is posted, a reporting mechanism with timely responses, the ability to block abusive users and published contact information. It also makes developers responsible for removing material that violates the rule, their own terms or their community standards.

An image tool's generation controls and its response to reported abuse therefore need to support the same moderation obligations. That is our reading of the combined rule and announcement. Apple has not prescribed a particular detection model or published a numerical accuracy threshold in these documents.

Hidden features have a separate rule

Apple's dated revision notice identifies 2.5.19 as a new numbered guideline clarifying a longstanding prohibition previously covered by 2.3.1. It addresses hidden, dormant or undocumented functionality.

Apple's accompanying explanation describes apps that conceal harmful capabilities during review and reveal them after approval, including tools that create sexual images without consent. The developer can face removal of the app and removal from the Apple Developer Program. This rule concerns what developers disclose to reviewers as well as the content their app produces.

The same revision notice says Section 5 clarifies removal and alternative-distribution blocking for apps that facilitate human trafficking or exploitation of children. That provision has its own scope; the update does not place every violation of Guideline 1.2 under that distribution block.

Apple describes remediation and removal

The current guidelines describe a possible remediation route: Apple may ask a developer to remove violating content and provide an improvement plan, then keep the app off the store until it demonstrates compliance. Egregious or repeated conduct can lead to immediate removal of both the app and developer.

BIG CHANGE reviewed Apple's announcement, revision notice and current rules; it did not test app safeguards or audit Apple's review decisions. The documents establish the policy change and stated consequences, without showing whether the revised wording reduces abuse.

Sources & further reading