diff --git a/docs/superpowers/plans/superpowers-add-email-to-personalid-signup.md b/docs/superpowers/plans/superpowers-add-email-to-personalid-signup.md new file mode 100644 index 0000000000..a38868947c --- /dev/null +++ b/docs/superpowers/plans/superpowers-add-email-to-personalid-signup.md @@ -0,0 +1,2833 @@ +# Add Email to PersonalID Signup / Recovery Flow — Implementation Plan + +> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Insert an optional email-entry + OTP-verification step into the PersonalID signup and recovery flows, positioned after Backup Code, **gated by a server-driven `email_otp_verification` toggle** returned from `/users/start_configuration`. During recovery, additionally skip the email step entirely when the server already returns a verified `email` for the user. Also offer a non-blocking email-collection prompt to existing users who completed signup without an email. + +**Architecture:** New Kotlin fragments (`PersonalIdEmailFragment`, `PersonalIdEmailVerificationFragment`) extend the existing `BasePersonalIdFragment` pattern. Both fragments accept an `isRecovery` nav arg (in addition to the existing `isLegacyFlow` arg). The Email/Email-OTP fragments branch on exit: signup continues to Photo Capture (which writes the record's email field from session data); recovery invokes a new `PersonalIdRecoveryCompleter` helper that runs the original recovery-finalization logic (DB passphrase, write record with email, recovery analytics, second-device notification) and then navigates to the existing recovery-success message destination. + +**Toggle-driven inclusion:** the `/users/start_configuration` response carries a `email_otp_verification` entry inside the existing `toggles` JSON object (NOT a separate top-level field) plus an optional top-level `email` (only when the user has already verified one server-side). The toggle is parsed by the existing `ConnectReleaseToggleRecord.releaseTogglesFromJson(json)` call already in `StartConfigurationResponseParser` — no new parsing code is needed for the toggle. The slug `"email_otp_verification"` lands in `sessionData.featureReleaseToggles` during the active PersonalID session and is persisted to the existing `connect_release_toggles` SQL table by `PersonalIdMessageFragment.successFlow.storeFeatureReleaseToggles()` after a successful PersonalID completion. Callers read it from two places depending on where they run: + +- **During the PersonalID flow** (e.g. `PersonalIdBackupCodeFragment`): read from `sessionData.featureReleaseToggles`. The DB hasn't been updated yet at this point. +- **Outside the PersonalID flow** (e.g. `PersonalIdManager.shouldOfferEmail()` called from `StandardHomeActivity.onCreate`): read from `ConnectAppDatabaseUtil.getReleaseToggles(context)`. Slug missing or `active = true` is permissive (legacy upgrade still sees the prompt); only an explicit `active = false` row suppresses the prompt. + +`PersonalIdBackupCodeFragment` decides where to go AFTER backup-code validation based on those values: + +- **Signup** (`email_otp_verification` toggle active): Phone → Biometrics → Phone OTP → Name → Backup Code → **Email (optional) → Email OTP** → Photo +- **Signup** (toggle inactive or slug missing): unchanged — Backup Code goes straight to Photo via the existing `navigateToPhoto()` +- **Recovery** (server returned `email`, regardless of toggle): skip the email step entirely — finalize recovery directly and show the recovery-success message +- **Recovery** (no server `email`, toggle active): Backup Code → Email (optional) → Email OTP (only if email entered) +- **Recovery** (no server `email`, toggle inactive or slug missing): unchanged — finalize recovery directly via `PersonalIdRecoveryCompleter` + `navigateWithMessage(...)` to the recovery-success destination + +In both fall-through cases (toggle off OR recovery-with-email) the backup-code fragment uses the pre-existing direct paths, which means **`navigateToPhoto()` and the matching `action_personalid_backupcode_to_personalid_photo_capture` action must be kept**, and `navigateWithMessage(...)` (already in the file) is reused for the no-email recovery success. + +**Persistence split:** only `email` is added to `ConnectUserRecord` (new DB column in v25). Three supporting flags — `emailVerified`, `emailOfferCount`, `lastEmailOfferDate` — live in a dedicated `personalid_prefs` SharedPreferences file accessed through a new `PersonalIDPreferences` Kotlin object. All three have null-aware semantics (absent key = "never set") and are wiped together on logout. The `email_otp_verification` toggle is NOT in `PersonalIDPreferences` — it lives in the existing `connect_release_toggles` SQL table (populated by the existing toggle pipeline). `PersonalIdManager.shouldOfferEmail()` reads the toggle from that table; an explicit `active = false` row suppresses the legacy prompt, while a missing-row (legacy user has never triggered start_configuration since upgrading) or `active = true` is treated as permissive so legacy users still see the offer. Existing users without a verified email are handled by the legacy prompt (two-offer-with-30-day-gap, gated on the toggle) read/written via `PersonalIDPreferences`. + +**Tech Stack:** Kotlin (new files), Java (existing files modified), AndroidX Navigation Safe Args, Retrofit 2 / OkHttp, SQLCipher (versioned migration), Robolectric unit tests. + +--- + +## File Structure + +### Files to Create +| File | Purpose | +|------|---------| +| `app/src/org/commcare/android/database/connect/models/ConnectUserRecordV24.java` | Snapshot of v24 ConnectUserRecord (for DB migration) | +| `app/src/org/commcare/fragments/personalId/PersonalIdEmailFragment.kt` | Email entry UI fragment | +| `app/src/org/commcare/fragments/personalId/PersonalIdEmailVerificationFragment.kt` | Email OTP verification UI fragment | +| `app/res/layout/fragment_personalid_email.xml` | Layout for email entry screen | +| `app/res/layout/fragment_personalid_email_verification.xml` | Layout for email OTP screen | +| `app/src/org/commcare/connect/network/connectId/parser/SendEmailOtpResponseParser.kt` | Parses send-email-OTP server response | +| `app/src/org/commcare/connect/network/connectId/parser/VerifyEmailOtpResponseParser.kt` | Parses verify-email-OTP server response | +| `app/src/org/commcare/connect/PersonalIdRecoveryCompleter.kt` | Helper holding the account-recovery finalization extracted from `PersonalIdBackupCodeFragment` (DB passphrase, write record, analytics, second-device notification) | +| `app/src/org/commcare/connect/PersonalIDPreferences.kt` | SharedPreferences wrapper holding `emailVerified`, `emailOfferCount`, `lastEmailOfferDate` with null-aware getters/setters and `clear()` for logout | +| `app/unit-tests/src/org/commcare/connect/PersonalIdEmailOfferTest.kt` | Unit tests for evaluateEmailOffer logic (Robolectric-backed, since prefs need a Context) | +| `app/unit-tests/src/org/commcare/connect/PersonalIDPreferencesTest.kt` | Unit tests for the prefs wrapper (null semantics, clear, round-trip) | +| `app/unit-tests/src/org/commcare/connect/PersonalIdEmailValidationTest.kt` | Unit tests for client-side email format validation | + +### Files to Modify +| File | Change | +|------|--------| +| `app/src/org/commcare/android/database/connect/models/ConnectUserRecord.java` | Add `email` field only (the other three flags live in `PersonalIDPreferences`) | +| `app/src/org/commcare/android/database/connect/models/PersonalIdSessionData.kt` | Add `email`, `emailSkippedDuringSignup` fields and a helper `isEmailOtpVerificationToggleActive(): Boolean` that scans `featureReleaseToggles` for slug `email_otp_verification`. `email != null` is the in-session source of truth for "verified" — no separate `emailVerified` field | +| `app/src/org/commcare/connect/network/connectId/parser/StartConfigurationResponseParser.java` | Parse the new top-level `email` field (when present, treat as already-verified). The `email_otp_verification` toggle is already parsed via the existing `ConnectReleaseToggleRecord.releaseTogglesFromJson(json)` call — no toggle-specific parsing code needed | +| `app/unit-tests/src/org/commcare/connect/network/connectId/parser/StartConfigurationResponseParserTest.kt` | Add two test cases for the new `email` field handling (present sets emailVerified=true, absent leaves emailVerified=false) | +| `app/src/org/commcare/fragments/personalId/PersonalIdPhotoCaptureFragment.java` | Read email/emailVerified from session data when constructing ConnectUserRecord (signup path only) | +| `app/src/org/commcare/fragments/personalId/PersonalIdBackupCodeFragment.java` | Store backup code on session data; branch on `sessionData.isEmailOtpVerificationToggleActive()` and (recovery only) `sessionData.email`. Signup with toggle active → `navigateToEmail(false)`; signup with toggle inactive → keep using `navigateToPhoto()`. Recovery with server email or toggle inactive → call new `completeRecoveryWithoutEmail()` helper (uses `PersonalIdRecoveryCompleter` + `navigateWithMessage(...)`); recovery with no server email + toggle active → `navigateToEmail(true)`. Extract finalization into `PersonalIdRecoveryCompleter`; delete `handleSuccessfulRecovery()`/`navigateToSuccess()`/`logRecoveryResult()`/`handleSecondDeviceLogin()`; KEEP `navigateToPhoto()` and `navigateWithMessage(...)`. | +| `app/src/org/commcare/models/database/connect/DatabaseConnectOpenHelper.java` | Bump CONNECT_DB_VERSION to 25 | +| `app/src/org/commcare/models/database/connect/ConnectDatabaseUpgrader.java` | Add upgradeTwentyFourTwentyFive() | +| `app/src/org/commcare/connect/network/ApiEndPoints.java` | Add sendEmailOtp and verifyEmailOtp endpoints | +| `app/src/org/commcare/connect/network/ApiService.java` | Add sendEmailOtp and verifyEmailOtp service methods | +| `app/src/org/commcare/connect/network/ApiPersonalId.java` | Add sendEmailOtp() and verifyEmailOtp() static methods | +| `app/src/org/commcare/connect/network/connectId/PersonalIdApiHandler.java` | Add sendEmailOtpCall() and verifyEmailOtpCall() | +| `app/res/navigation/nav_graph_personalid.xml` | Add email and email-OTP destinations (with `isLegacyFlow` and `isRecovery` args); ADD `action_personalid_backupcode_to_personalid_email` (do NOT remove the existing `..._photo_capture` or `..._message` actions — both are still used by the toggle-off / email-already-verified branches); add Email/Email-OTP → PhotoCapture (signup) and Email/Email-OTP → MessageDisplay (recovery success) | +| `app/src/org/commcare/activities/connect/PersonalIdActivity.java` | Handle EXTRA_LEGACY_EMAIL_FLOW intent extra | +| `app/src/org/commcare/connect/PersonalIdManager.java` | Add checkEmailCollection() and shouldOfferEmail() — both read/write via `PersonalIDPreferences`. Also call `PersonalIDPreferences.clear(context)` in the PersonalID logout path (see Task 8b). | +| `app/src/org/commcare/activities/StandardHomeActivity.java` | Call checkEmailCollection() in `onCreate` — single entry point for the legacy prompt, fires on every CommCare app open | + +--- + +## Chunk 1: Database Layer + +### Task 1: Create ConnectUserRecordV24 (migration snapshot) + +**Files:** +- Create: `app/src/org/commcare/android/database/connect/models/ConnectUserRecordV24.java` + +- [ ] **Step 1.1: Write the failing unit test** + + Create `app/unit-tests/src/org/commcare/connect/ConnectUserRecordMigrationV24Test.kt`: + + ```kotlin + package org.commcare.connect + + import org.junit.Assert.assertNull + import org.junit.Test + + class ConnectUserRecordMigrationV24Test { + + @Test + fun `fromV24 copies all fields and sets email to null`() { + val old = ConnectUserRecordV24().apply { + // Verify V24 compiles with fields 1-16 only (no email fields) + } + val new = ConnectUserRecord.fromV24(old) + assertNull(new.email) + } + } + ``` + + > **Note:** `emailVerified`, `emailOfferCount`, and `lastEmailOfferDate` are NOT fields on `ConnectUserRecord` — they live in `PersonalIDPreferences` (added in Task 3b) and are not touched by this migration. + + Run: `./gradlew testCommcareDebug --tests "org.commcare.connect.ConnectUserRecordMigrationV24Test"` + Expected: FAIL — `ConnectUserRecordV24` and `ConnectUserRecord.fromV24` don't exist yet. + +- [ ] **Step 1.2: Create ConnectUserRecordV24.java** + + This is a verbatim copy of the current `ConnectUserRecord.java` with the class name changed and `STORAGE_KEY` kept. It will only ever be read, not written. + + ```java + package org.commcare.android.database.connect.models; + + import org.commcare.android.storage.framework.Persisted; + import org.commcare.connect.ConnectConstants; + import org.commcare.models.framework.Persisting; + import org.commcare.modern.database.Table; + import org.commcare.modern.models.MetaField; + import java.util.Date; + + @Table(ConnectUserRecordV24.STORAGE_KEY) + public class ConnectUserRecordV24 extends Persisted { + public static final String STORAGE_KEY = "user_info"; + public static final String META_PIN = "pin"; + + @Persisting(1) private String userId; + @Persisting(2) private String password; + @Persisting(3) private String name; + @Persisting(4) private String primaryPhone; + @Deprecated @Persisting(5) private String alternatePhone; + @Persisting(6) private int registrationPhase; + @Persisting(7) private Date lastPasswordDate; + @Persisting(value = 8, nullable = true) private String connectToken; + @Persisting(value = 9, nullable = true) private Date connectTokenExpiration; + @Persisting(value = 10, nullable = true) @MetaField(META_PIN) private String pin; + @Deprecated @Persisting(11) private boolean secondaryPhoneVerified; + @Deprecated @Persisting(12) private Date verifySecondaryPhoneByDate; + @Persisting(value = 13, nullable = true) private String photo; + @Persisting(value = 14) private boolean isDemo; + @Persisting(value = 15) private String requiredLock = PersonalIdSessionData.PIN; + @Persisting(value = 16) private boolean hasConnectAccess; + + public ConnectUserRecordV24() { + registrationPhase = ConnectConstants.PERSONALID_NO_ACTIVITY; + lastPasswordDate = new Date(); + connectTokenExpiration = new Date(); + secondaryPhoneVerified = true; + verifySecondaryPhoneByDate = new Date(); + alternatePhone = ""; + } + + // Getters for all fields (copy from ConnectUserRecord) + public String getUserId() { return userId; } + public String getPassword() { return password; } + public String getName() { return name; } + public String getPrimaryPhone() { return primaryPhone; } + public int getRegistrationPhase() { return registrationPhase; } + public Date getLastPasswordDate() { return lastPasswordDate; } + public String getConnectToken() { return connectToken; } + public Date getConnectTokenExpiration() { return connectTokenExpiration; } + public String getPin() { return pin; } + public String getPhoto() { return photo; } + public boolean isDemo() { return isDemo; } + public String getRequiredLock() { return requiredLock; } + public boolean hasConnectAccess() { return hasConnectAccess; } + } + ``` + +- [ ] **Step 1.3: Add email field and fromV24() to ConnectUserRecord.java** + + In `ConnectUserRecord.java`, after field `@Persisting(value = 16)`: + + ```java + public static final String META_EMAIL = "email"; + + @Persisting(value = 17, nullable = true) + @MetaField(META_EMAIL) + private String email; + ``` + + Add getter and setter: + + ```java + public String getEmail() { return email; } + public void setEmail(String email) { this.email = email; } + ``` + + > **Note:** `emailVerified`, `emailOfferCount`, and `lastEmailOfferDate` are NOT fields on `ConnectUserRecord`. They live in `PersonalIDPreferences` (Task 3b) — do not add them here. + + Add the migration factory method: + + ```java + public static ConnectUserRecord fromV24(ConnectUserRecordV24 oldRecord) { + ConnectUserRecord newRecord = new ConnectUserRecord(); + newRecord.userId = oldRecord.getUserId(); + newRecord.password = oldRecord.getPassword(); + newRecord.name = oldRecord.getName(); + newRecord.primaryPhone = oldRecord.getPrimaryPhone(); + newRecord.alternatePhone = ""; + newRecord.registrationPhase = oldRecord.getRegistrationPhase(); + newRecord.lastPasswordDate = oldRecord.getLastPasswordDate(); + newRecord.connectToken = oldRecord.getConnectToken(); + newRecord.connectTokenExpiration = oldRecord.getConnectTokenExpiration(); + newRecord.secondaryPhoneVerified = true; + newRecord.photo = oldRecord.getPhoto(); + newRecord.isDemo = oldRecord.isDemo(); + newRecord.requiredLock = oldRecord.getRequiredLock(); + newRecord.hasConnectAccess = oldRecord.hasConnectAccess(); + // email defaults to null (the other flags live in PersonalIDPreferences, not this record) + return newRecord; + } + ``` + +- [ ] **Step 1.4: Run the test to verify it passes** + + Run: `./gradlew testCommcareDebug --tests "org.commcare.connect.ConnectUserRecordMigrationV24Test"` + Expected: PASS + +- [ ] **Step 1.5: Commit** + + ```bash + git add app/src/org/commcare/android/database/connect/models/ConnectUserRecordV24.java \ + app/src/org/commcare/android/database/connect/models/ConnectUserRecord.java \ + app/unit-tests/src/org/commcare/connect/ConnectUserRecordMigrationV24Test.kt + git commit -m "[AI] Add email field to ConnectUserRecord with V24 snapshot for migration" + ``` + +--- + +### Task 2: Add DB migration v24 → v25 + +**Files:** +- Modify: `app/src/org/commcare/models/database/connect/DatabaseConnectOpenHelper.java` +- Modify: `app/src/org/commcare/models/database/connect/ConnectDatabaseUpgrader.java` + +- [ ] **Step 2.1: Bump CONNECT_DB_VERSION to 25** + + In `DatabaseConnectOpenHelper.java`, line 70: + + ```java + // Before: + private static final int CONNECT_DB_VERSION = 24; + + // After: + private static final int CONNECT_DB_VERSION = 25; + ``` + + Also add a version comment in the header block: + ```java + // V.25 - Added email column to ConnectUserRecord (emailVerified / offerCount / lastOfferDate live in PersonalIDPreferences, not the DB) + ``` + +- [ ] **Step 2.2: Add import for ConnectUserRecordV24 in ConnectDatabaseUpgrader.java** + + In `ConnectDatabaseUpgrader.java`, in the imports block, add: + ```java + import org.commcare.android.database.connect.models.ConnectUserRecordV24; + ``` + +- [ ] **Step 2.3: Add upgradeTwentyFourTwentyFive() to ConnectDatabaseUpgrader.java** + + First, add the call in the `upgrade()` method after the existing v23→v24 block: + + ```java + if (oldVersion == 24) { + upgradeTwentyFourTwentyFive(db); + oldVersion = 25; + } + ``` + + Then add the method: + + ```java + private void upgradeTwentyFourTwentyFive(IDatabase db) { + db.beginTransaction(); + try { + DbUtil.addColumnToTable(db, ConnectUserRecord.STORAGE_KEY, + ConnectUserRecord.META_EMAIL, "TEXT"); + + // Migrate existing records: read V24, write new ConnectUserRecord + SqlStorage oldStorage = new SqlStorage<>( + ConnectUserRecordV24.STORAGE_KEY, ConnectUserRecordV24.class, + new ConcreteAndroidDbHelper(c, db)); + SqlStorage newStorage = new SqlStorage<>( + ConnectUserRecord.STORAGE_KEY, ConnectUserRecord.class, + new ConcreteAndroidDbHelper(c, db)); + + for (ConnectUserRecordV24 old : oldStorage) { + ConnectUserRecord newRecord = ConnectUserRecord.fromV24(old); + newRecord.setID(old.getID()); + newStorage.write(newRecord); + } + db.setTransactionSuccessful(); + } catch (Exception e) { + CrashUtil.reportException(e); + } finally { + db.endTransaction(); + } + } + ``` + + > **Note:** Check how `DbUtil.addColumnToTable(IDatabase, String, String, String)` is called in other migration methods in this file and match the exact signature. Some versions take the db object directly; others call it differently. + +- [ ] **Step 2.4: Build to verify compilation** + + Run: `./gradlew assembleCommcareDebug` + Expected: BUILD SUCCESSFUL + +- [ ] **Step 2.5: Commit** + + ```bash + git add app/src/org/commcare/models/database/connect/DatabaseConnectOpenHelper.java \ + app/src/org/commcare/models/database/connect/ConnectDatabaseUpgrader.java + git commit -m "[AI] Add DB migration v24→v25 adding email column to ConnectUserRecord" + ``` + +--- + +## Chunk 2: Session Data & API Layer + +### Task 3: Add email to PersonalIdSessionData + +**Files:** +- Modify: `app/src/org/commcare/android/database/connect/models/PersonalIdSessionData.kt` + +- [ ] **Step 3.1: Add email properties and toggle helper** + + Open `PersonalIdSessionData.kt` and add to the data class (alongside the existing fields, e.g., after `userName`): + + ```kotlin + var email: String? = null + var emailSkippedDuringSignup: Boolean = false + ``` + + > **No `emailVerified` field.** During the PersonalID session, `email != null` is the source of truth for "this user has a verified email." Producers (parser, OTP-verify success) set the field; consumers (PhotoCapture, RecoveryCompleter) read `email != null` to decide what to write to `PersonalIDPreferences.emailVerified`. + + > **Only two places ever assign `sessionData.email`:** (1) `StartConfigurationResponseParser` writes it from the response (null when omitted, the verified address when returned), and (2) `PersonalIdEmailVerificationFragment.onEmailVerified()` writes the just-verified address after a successful `verify_email_otp` call. **Nothing else mutates this field** — `sendEmailOtpCall` does not write to it (the entered-but-not-yet-verified address rides as a nav arg through the OTP fragment), and the skip / proceed-without-email paths do not null it out (the field is already null at that point because nothing has ever written to it in this flow). This is by design — wider mutation would let unverified addresses leak into `ConnectUserRecord.email` if the user backed out at the wrong moment. + + Add a companion-object constant for the toggle slug and a helper method on the class: + + ```kotlin + companion object { + // existing constants stay above + const val EMAIL_OTP_VERIFICATION_SLUG = "email_otp_verification" + } + + /** + * True when the `email_otp_verification` slug is present in the parsed + * featureReleaseToggles list AND its `active` flag is true. Returns false when the + * slug is missing OR explicitly false. Callers running OUTSIDE the PersonalID flow + * (e.g. the legacy prompt) MUST instead read from ConnectAppDatabaseUtil — see + * PersonalIdManager.shouldOfferEmail() — because featureReleaseToggles is only + * populated during an active PersonalID session. + */ + fun isEmailOtpVerificationToggleActive(): Boolean = + featureReleaseToggles?.firstOrNull { it.slug == EMAIL_OTP_VERIFICATION_SLUG }?.active == true + ``` + + > **NO new field for the toggle.** The `email_otp_verification` toggle is delivered as a slug inside the existing `toggles` JSON object on the `/users/start_configuration` response and is already parsed into `featureReleaseToggles` by the existing `ConnectReleaseToggleRecord.releaseTogglesFromJson(json)` call in `StartConfigurationResponseParser`. We just need to read the slug from that list — no parser changes required for the toggle itself. + + > **`email` semantics during recovery:** `/users/start_configuration` returns the new top-level `email` field ONLY when the user has already verified an address server-side. When present, recovery treats it as ground truth — the email step is skipped, and `PersonalIdRecoveryCompleter` writes the value to `ConnectUserRecord.email` and writes `PersonalIDPreferences.emailVerified = (sessionData.email != null)` (true in this case). Task 3a updates the parser to set `sessionData.email` from the response. + + > **Why these live in session data:** Neither signup nor recovery has written `ConnectUserRecord` by the time the user acts on the email screen. In signup, `PersonalIdPhotoCaptureFragment` writes the record later. In recovery, `PersonalIdBackupCodeFragment` used to write it immediately on successful backup-code validation, but in the new flow that write is deferred — it now happens in `PersonalIdRecoveryCompleter` after the email step. Session data is the carrier in both cases. (Backup code is also now stored on session data so `PersonalIdRecoveryCompleter` can read it after the email step.) + + > **Why `emailSkippedDuringSignup`:** A user who explicitly skips email during signup has already been presented the offer once. If `emailOfferCount` (in `PersonalIDPreferences`) were left at its default-absent value, `shouldOfferEmail()` would show the dialog again on the very next login, immediately after they just declined during signup. When this flag is set, the PhotoCapture / RecoveryCompleter step writes `emailOfferCount = 1` and `lastEmailOfferDate = now` into `PersonalIDPreferences`, treating the signup screen as the first offer so the dialog only appears 30 days later (second and final offer). Legacy users migrated from v24 have never-set prefs keys (i.e. `emailOfferCount` reads as `null`), which `shouldOfferEmail()` treats as "never offered" — their first dialog appears on the next login as intended. + > + > **Recovery-skip behaviour:** The same rule applies when the user skips email during the recovery flow — `PersonalIdRecoveryCompleter` reads `emailSkippedDuringSignup` and writes `emailOfferCount = 1` + `lastEmailOfferDate = now` into `PersonalIDPreferences` after finalizing recovery. Recovery-skip is product-equivalent to signup-skip (the user was shown the offer once and declined), so the legacy prompt should not re-fire for 30 days. Confirm with product that this is the desired behaviour during recovery before shipping; if it is not, remove the `emailSkippedDuringSignup` handling from `PersonalIdRecoveryCompleter.finalizeAccountRecovery()` (Task 6c.2) and update the manual-test expectation in Step 9.3 case 3b accordingly. + +- [ ] **Step 3.2: Build to verify compilation** + + Run: `./gradlew assembleCommcareDebug` + Expected: BUILD SUCCESSFUL + +- [ ] **Step 3.3: Commit** + + ```bash + git add app/src/org/commcare/android/database/connect/models/PersonalIdSessionData.kt + git commit -m "[AI] Add email field to PersonalIdSessionData" + ``` + +--- + +### Task 3a: Parse the new top-level `email` from `/users/start_configuration` + +The `/users/start_configuration` response now includes an optional `email` string when the user already has a verified email server-side. The `email_otp_verification` toggle is delivered inside the existing `toggles` JSON object and is already parsed by `ConnectReleaseToggleRecord.releaseTogglesFromJson(json)` — no toggle-parsing changes are required in this task. + +**Files:** +- Modify: `app/src/org/commcare/connect/network/connectId/parser/StartConfigurationResponseParser.java` +- Modify: `app/unit-tests/src/org/commcare/connect/network/connectId/parser/StartConfigurationResponseParserTest.kt` (existing file; add two new test cases) + +- [ ] **Step 3a.1: Add failing unit tests to the existing parser test** + + The test file `app/unit-tests/src/org/commcare/connect/network/connectId/parser/StartConfigurationResponseParserTest.kt` already exists with `@RunWith(AndroidJUnit4::class)` + `@Config(application = CommCareTestApplication::class)` (Robolectric). Append the following two `@Test` methods inside the existing class — match the existing camelCase test-name style: + + ```kotlin + @Test + fun testParseSetsEmailWhenServerReturnsIt() { + val json = JSONObject().apply { + put("email", "user@example.com") + } + + parser.parse(json, sessionData) + + assertEquals("user@example.com", sessionData.email) + } + + @Test + fun testParseLeavesEmailNullWhenServerOmitsEmail() { + val json = JSONObject() + + parser.parse(json, sessionData) + + assertNull(sessionData.email) + } + ``` + + No new imports required — `JSONObject`, `assertEquals`, `assertNull`, and `Test` are all already imported by the existing file. + + Run: `./gradlew testCommcareDebug --tests "org.commcare.connect.network.connectId.parser.StartConfigurationResponseParserTest"` + Expected: the two new tests FAIL — parser does not yet read `email`. Existing tests should still pass. + +- [ ] **Step 3a.2: Update StartConfigurationResponseParser.java** + + Add the following inside `parse(...)`, right after the existing `setOtpFallback(...)` line (and before the `featureReleaseToggles` block, which is unchanged): + + ```java + // Email is returned ONLY when the server already has a verified address for this user. + // When present, the recovery flow can skip the email step (see PersonalIdBackupCodeFragment). + // Setting sessionData.email is itself the "verified" signal for downstream consumers — + // there is no separate emailVerified field. + sessionData.setEmail(JsonExtensions.optStringSafe(json, "email", null)); + ``` + + No constructor signature change. No callsite changes. No PersonalIDPreferences write — the toggle is persisted by the existing `connect_release_toggles` pipeline (via `PersonalIdMessageFragment.successFlow.storeFeatureReleaseToggles()`), which we do not touch. + + > **Note:** `setEmail` is the auto-generated Kotlin setter from Task 3.1 (a data-class `var` property produces a Java setter automatically — no extra `@JvmField` annotation required). + +- [ ] **Step 3a.3: Run the parser tests to verify they pass** + + Run: `./gradlew testCommcareDebug --tests "org.commcare.connect.network.connectId.parser.StartConfigurationResponseParserTest"` + Expected: PASS — the two new tests now succeed and existing tests remain green. + +- [ ] **Step 3a.4: Commit** + + ```bash + git add app/src/org/commcare/connect/network/connectId/parser/StartConfigurationResponseParser.java \ + app/unit-tests/src/org/commcare/connect/network/connectId/parser/StartConfigurationResponseParserTest.kt + git commit -m "[AI] Parse top-level email from start_configuration response" + ``` + +--- + +### Task 3b: Create PersonalIDPreferences (SharedPreferences wrapper for the three email flags) + +Only `email` is persisted on `ConnectUserRecord`. The three supporting flags — `emailVerified`, `emailOfferCount`, `lastEmailOfferDate` — live in a dedicated `personalid_prefs` SharedPreferences file accessed through a Kotlin `object` wrapper. Tasks 6c, 6c.2, 8, and 8b all depend on this class, so it must exist before they run. + +> **Why the toggle is NOT here:** the `email_otp_verification` toggle is already persisted by the existing `connect_release_toggles` SQL table populated via `ConnectAppDatabaseUtil.storeReleaseToggles(...)`. `PersonalIdManager.shouldOfferEmail()` (Task 8) reads it from there using `ConnectAppDatabaseUtil.getReleaseToggles(context)`. Duplicating it into prefs would create a second source of truth. + +**Files:** +- Create: `app/src/org/commcare/connect/PersonalIDPreferences.kt` +- Create: `app/unit-tests/src/org/commcare/connect/PersonalIDPreferencesTest.kt` + +- [ ] **Step 3b.1: Write the failing unit test** + + Create `app/unit-tests/src/org/commcare/connect/PersonalIDPreferencesTest.kt`: + + ```kotlin + package org.commcare.connect + + import androidx.test.core.app.ApplicationProvider + import org.junit.After + import org.junit.Assert.assertEquals + import org.junit.Assert.assertNull + import org.junit.Test + import org.junit.runner.RunWith + import org.robolectric.RobolectricTestRunner + import java.util.Date + + @RunWith(RobolectricTestRunner::class) + class PersonalIDPreferencesTest { + + private val context get() = ApplicationProvider.getApplicationContext() + + @After + fun tearDown() { + PersonalIDPreferences.clear(context) + } + + @Test + fun `all getters return null when prefs are empty`() { + assertNull(PersonalIDPreferences.isEmailVerified(context)) + assertNull(PersonalIDPreferences.getEmailOfferCount(context)) + assertNull(PersonalIDPreferences.getLastEmailOfferDate(context)) + } + + @Test + fun `round-trips typed values`() { + val when_ = Date() + PersonalIDPreferences.setEmailVerified(context, true) + PersonalIDPreferences.setEmailOfferCount(context, 2) + PersonalIDPreferences.setLastEmailOfferDate(context, when_) + + assertEquals(true, PersonalIDPreferences.isEmailVerified(context)) + assertEquals(2, PersonalIDPreferences.getEmailOfferCount(context)) + assertEquals(when_.time, PersonalIDPreferences.getLastEmailOfferDate(context)?.time) + } + + @Test + fun `setter with null removes the key`() { + PersonalIDPreferences.setEmailVerified(context, true) + PersonalIDPreferences.setEmailVerified(context, null) + assertNull(PersonalIDPreferences.isEmailVerified(context)) + } + + @Test + fun `clear removes all keys`() { + PersonalIDPreferences.setEmailVerified(context, true) + PersonalIDPreferences.setEmailOfferCount(context, 1) + PersonalIDPreferences.setLastEmailOfferDate(context, Date()) + + PersonalIDPreferences.clear(context) + + assertNull(PersonalIDPreferences.isEmailVerified(context)) + assertNull(PersonalIDPreferences.getEmailOfferCount(context)) + assertNull(PersonalIDPreferences.getLastEmailOfferDate(context)) + } + + @Test + fun `explicitly-stored false is distinguishable from unset`() { + PersonalIDPreferences.setEmailVerified(context, false) + // contains() must be true, so getter returns false (not null) + assertEquals(false, PersonalIDPreferences.isEmailVerified(context)) + } + } + ``` + + Run: `./gradlew testCommcareDebug --tests "org.commcare.connect.PersonalIDPreferencesTest"` + Expected: FAIL — `PersonalIDPreferences` does not exist yet. + +- [ ] **Step 3b.2: Create PersonalIDPreferences.kt** + + Create `app/src/org/commcare/connect/PersonalIDPreferences.kt`: + + ```kotlin + package org.commcare.connect + + import android.content.Context + import android.content.SharedPreferences + import java.util.Date + + /** + * SharedPreferences-backed store for the three flags that accompany `ConnectUserRecord.email`: + * - emailVerified (Boolean) + * - emailOfferCount (Int) — 0 = never offered, 1 = first offer shown, 2 = both offers shown + * - lastEmailOfferDate (Date) — when the most recent offer was shown + * + * The `email_otp_verification` toggle is NOT stored here — it lives in the existing + * `connect_release_toggles` SQL table (see ConnectAppDatabaseUtil.getReleaseToggles). + * + * Null-aware: an absent key returns null (not a false/0/0L default). Callers must handle null + * (typically by treating it as "never set", equivalent to a freshly-migrated v24 user). + * + * On PersonalID logout, call [clear] to wipe every key in this prefs file. + */ + object PersonalIDPreferences { + + private const val PREFS_NAME = "personalid_prefs" + private const val KEY_EMAIL_VERIFIED = "email_verified" + private const val KEY_EMAIL_OFFER_COUNT = "email_offer_count" + private const val KEY_LAST_EMAIL_OFFER_DATE = "last_email_offer_date" + + private fun prefs(context: Context): SharedPreferences = + context.applicationContext.getSharedPreferences(PREFS_NAME, Context.MODE_PRIVATE) + + @JvmStatic + fun isEmailVerified(context: Context): Boolean? { + val p = prefs(context) + return if (p.contains(KEY_EMAIL_VERIFIED)) p.getBoolean(KEY_EMAIL_VERIFIED, false) else null + } + + @JvmStatic + fun setEmailVerified(context: Context, value: Boolean?) { + prefs(context).edit().apply { + if (value == null) remove(KEY_EMAIL_VERIFIED) else putBoolean(KEY_EMAIL_VERIFIED, value) + }.apply() + } + + @JvmStatic + fun getEmailOfferCount(context: Context): Int? { + val p = prefs(context) + return if (p.contains(KEY_EMAIL_OFFER_COUNT)) p.getInt(KEY_EMAIL_OFFER_COUNT, 0) else null + } + + @JvmStatic + fun setEmailOfferCount(context: Context, value: Int?) { + prefs(context).edit().apply { + if (value == null) remove(KEY_EMAIL_OFFER_COUNT) else putInt(KEY_EMAIL_OFFER_COUNT, value) + }.apply() + } + + @JvmStatic + fun getLastEmailOfferDate(context: Context): Date? { + val p = prefs(context) + return if (p.contains(KEY_LAST_EMAIL_OFFER_DATE)) Date(p.getLong(KEY_LAST_EMAIL_OFFER_DATE, 0L)) else null + } + + @JvmStatic + fun setLastEmailOfferDate(context: Context, value: Date?) { + prefs(context).edit().apply { + if (value == null) remove(KEY_LAST_EMAIL_OFFER_DATE) else putLong(KEY_LAST_EMAIL_OFFER_DATE, value.time) + }.apply() + } + + /** Remove every PersonalID preference. Call on logout. */ + @JvmStatic + fun clear(context: Context) { + prefs(context).edit().clear().apply() + } + } + ``` + + > **Why a dedicated prefs file (`personalid_prefs`) instead of `HiddenPreferences` or app prefs:** `clear()` does `edit().clear().apply()`, which only wipes keys in the file we own. Sharing app-level prefs would force enumerating each key individually and risk deleting unrelated state at logout. + +- [ ] **Step 3b.3: Run the tests to verify they pass** + + Run: `./gradlew testCommcareDebug --tests "org.commcare.connect.PersonalIDPreferencesTest"` + Expected: PASS (all 5 tests). + +- [ ] **Step 3b.4: Commit** + + ```bash + git add app/src/org/commcare/connect/PersonalIDPreferences.kt \ + app/unit-tests/src/org/commcare/connect/PersonalIDPreferencesTest.kt + git commit -m "[AI] Add PersonalIDPreferences wrapper for emailVerified/offerCount/lastOfferDate" + ``` + +--- + +### Task 4: Add email API endpoints + +> **Important:** The server-side endpoints `/users/send_email_otp` and `/users/verify_email_otp` are provisional. Confirm actual endpoint paths with the backend team before finalising. The parameter names `email` and `otp` are also provisional. + +**Files:** +- Modify: `app/src/org/commcare/connect/network/ApiEndPoints.java` +- Modify: `app/src/org/commcare/connect/network/ApiService.java` +- Modify: `app/src/org/commcare/connect/network/ApiPersonalId.java` +- Modify: `app/src/org/commcare/connect/network/connectId/PersonalIdApiHandler.java` +- Create: `app/src/org/commcare/connect/network/connectId/parser/SendEmailOtpResponseParser.kt` +- Create: `app/src/org/commcare/connect/network/connectId/parser/VerifyEmailOtpResponseParser.kt` + +- [ ] **Step 4.1: Write failing unit test for evaluateEmailOffer** + + Create `app/unit-tests/src/org/commcare/connect/PersonalIdEmailOfferTest.kt`: + + ```kotlin + package org.commcare.connect + + import android.content.Context + import androidx.test.core.app.ApplicationProvider + import org.junit.After + import org.junit.Assert.assertFalse + import org.junit.Assert.assertTrue + import org.junit.Test + import org.junit.runner.RunWith + import org.robolectric.RobolectricTestRunner + import java.util.Calendar + import java.util.Date + + @RunWith(RobolectricTestRunner::class) + class PersonalIdEmailOfferTest { + + private val context: Context get() = ApplicationProvider.getApplicationContext() + + @After + fun tearDown() { + PersonalIDPreferences.clear(context) + } + + private fun dateMinusDays(days: Int): Date = + Calendar.getInstance().apply { add(Calendar.DAY_OF_YEAR, -days) }.time + + // Helper: write a fake ConnectReleaseToggleRecord row matching the production + // shape of `connect_release_toggles`. Pass `null` to leave the table empty + // (legacy-upgrade scenario where the slug has never been seen). + private fun setToggleActive(active: Boolean?) { + if (active == null) return + val toggle = ConnectReleaseToggleRecord.releaseToggleFromJson( + PersonalIdSessionData.EMAIL_OTP_VERIFICATION_SLUG, + JSONObject().apply { + put("active", active) + put("created_at", "2026-01-01T00:00:00Z") + put("modified_at", "2026-01-01T00:00:00Z") + } + ) + ConnectAppDatabaseUtil.storeReleaseToggles(context, listOf(toggle)) + } + + @Test + fun `should offer when toggle slug missing (legacy upgrade)`() { + // connect_release_toggles is empty — start_configuration has not been parsed yet on + // this client. Permissive default: legacy users still see the offer on first + // app-open. A subsequent start_configuration parse with active=false will replace + // the missing row with an explicit false and suppress future prompts. + assertTrue(PersonalIdManager.shouldOfferEmail(context)) + } + + @Test + fun `should not offer when toggle is explicitly disabled`() { + setToggleActive(false) + assertFalse(PersonalIdManager.shouldOfferEmail(context)) + } + + @Test + fun `should offer when never shown before (prefs empty, toggle on)`() { + setToggleActive(true) + // emailVerified=null, offerCount=null, lastOfferDate=null — a legacy v24 user. + assertTrue(PersonalIdManager.shouldOfferEmail(context)) + } + + @Test + fun `should not offer when both offers already shown (count=2)`() { + setToggleActive(true) + PersonalIDPreferences.setEmailVerified(context, false) + PersonalIDPreferences.setEmailOfferCount(context, 2) + PersonalIDPreferences.setLastEmailOfferDate(context, dateMinusDays(5)) + assertFalse(PersonalIdManager.shouldOfferEmail(context)) + } + + @Test + fun `should offer second time after 30 days (count=1, date old)`() { + setToggleActive(true) + PersonalIDPreferences.setEmailVerified(context, false) + PersonalIDPreferences.setEmailOfferCount(context, 1) + PersonalIDPreferences.setLastEmailOfferDate(context, dateMinusDays(31)) + assertTrue(PersonalIdManager.shouldOfferEmail(context)) + } + + @Test + fun `should not offer second time before 30 days (count=1, date recent)`() { + setToggleActive(true) + PersonalIDPreferences.setEmailVerified(context, false) + PersonalIDPreferences.setEmailOfferCount(context, 1) + PersonalIDPreferences.setLastEmailOfferDate(context, dateMinusDays(15)) + assertFalse(PersonalIdManager.shouldOfferEmail(context)) + } + + @Test + fun `should not offer when email already verified`() { + setToggleActive(true) + PersonalIDPreferences.setEmailVerified(context, true) + assertFalse(PersonalIdManager.shouldOfferEmail(context)) + } + + @Test + fun `should not offer immediately after signup skip (count=1, date just set)`() { + setToggleActive(true) + // Simulates a user who skipped email during signup: count=1, date=now + PersonalIDPreferences.setEmailVerified(context, false) + PersonalIDPreferences.setEmailOfferCount(context, 1) + PersonalIDPreferences.setLastEmailOfferDate(context, Date()) + assertFalse(PersonalIdManager.shouldOfferEmail(context)) + } + } + ``` + + Run: `./gradlew testCommcareDebug --tests "org.commcare.connect.PersonalIdEmailOfferTest"` + Expected: FAIL — `PersonalIdManager.shouldOfferEmail(Context)` does not exist yet. + +- [ ] **Step 4.2: Add email API endpoints to ApiEndPoints.java** + + ```java + public static final String sendEmailOtp = "/users/send_email_otp"; + public static final String verifyEmailOtp = "/users/verify_email_otp"; + ``` + +- [ ] **Step 4.3: Add email service methods to ApiService.java** + + ```java + @POST(ApiEndPoints.sendEmailOtp) + Call sendEmailOtp(@Header("Authorization") String token, + @Body Map emailRequest); + + @POST(ApiEndPoints.verifyEmailOtp) + Call verifyEmailOtp(@Header("Authorization") String token, + @Body Map otpRequest); + ``` + +- [ ] **Step 4.4: Create SendEmailOtpResponseParser.kt** + + ```kotlin + package org.commcare.connect.network.connectId.parser + + import org.commcare.android.database.connect.models.PersonalIdSessionData + import org.json.JSONObject + + class SendEmailOtpResponseParser : PersonalIdApiResponseParser { + override fun parse(json: JSONObject, sessionData: PersonalIdSessionData) { + // Server may return additional info; extend as needed when API is finalised. + } + } + ``` + +- [ ] **Step 4.5: Create VerifyEmailOtpResponseParser.kt** + + ```kotlin + package org.commcare.connect.network.connectId.parser + + import org.commcare.android.database.connect.models.PersonalIdSessionData + import org.json.JSONObject + + class VerifyEmailOtpResponseParser : PersonalIdApiResponseParser { + override fun parse(json: JSONObject, sessionData: PersonalIdSessionData) { + // Confirmation only; extend when server API is finalised. + } + } + ``` + +- [ ] **Step 4.6: Add static methods to ApiPersonalId.java** + + ```java + public static void sendEmailOtp(Context context, String email, String token, IApiCallback callback) { + AuthInfo authInfo = new AuthInfo.TokenAuth(token); + String tokenAuth = HttpUtils.getCredential(authInfo); + Objects.requireNonNull(tokenAuth); + + HashMap params = new HashMap<>(); + params.put("email", email); + + ApiService apiService = PersonalIdApiClient.getClientApi(); + Call call = apiService.sendEmailOtp(tokenAuth, params); + BaseApi.Companion.callApi(context, call, callback, ApiEndPoints.sendEmailOtp); + } + + public static void verifyEmailOtp(Context context, String email, String otp, + String token, IApiCallback callback) { + AuthInfo authInfo = new AuthInfo.TokenAuth(token); + String tokenAuth = HttpUtils.getCredential(authInfo); + Objects.requireNonNull(tokenAuth); + + HashMap params = new HashMap<>(); + params.put("email", email); + params.put("otp", otp); + + ApiService apiService = PersonalIdApiClient.getClientApi(); + Call call = apiService.verifyEmailOtp(tokenAuth, params); + BaseApi.Companion.callApi(context, call, callback, ApiEndPoints.verifyEmailOtp); + } + ``` + +- [ ] **Step 4.7: Add handler methods to PersonalIdApiHandler.java** + + Add imports for new parsers at top of file: + ```java + import org.commcare.connect.network.connectId.parser.SendEmailOtpResponseParser; + import org.commcare.connect.network.connectId.parser.VerifyEmailOtpResponseParser; + ``` + + Add handler methods. **Neither method mutates `sessionData.email`** — the entered email is passed in as a parameter and routed straight to the API. `sessionData.email` is reserved for the verified address only (see Task 6.2 for the actual write site). + + ```java + public void sendEmailOtpCall(Activity activity, String email, PersonalIdSessionData sessionData) { + ApiPersonalId.sendEmailOtp( + activity, + email, + sessionData.getToken(), + createCallback(sessionData, new SendEmailOtpResponseParser()) + ); + } + + public void verifyEmailOtpCall(Activity activity, String email, String otp, + PersonalIdSessionData sessionData) { + ApiPersonalId.verifyEmailOtp( + activity, + email, + otp, + sessionData.getToken(), + createCallback(sessionData, new VerifyEmailOtpResponseParser()) + ); + } + ``` + + > **Critical:** `PersonalIdApiHandler` has two distinct `createCallback` overloads. Use the **private** overload defined directly in `PersonalIdApiHandler` — `createCallback(PersonalIdSessionData, PersonalIdApiResponseParser)`. Do NOT use the parent `BaseApiHandler.createCallback`, which takes a different parser interface and does not populate `sessionData`. Both compile without error but the parent overload leaves session data unpopulated. + +- [ ] **Step 4.8: Build to verify compilation** + + Run: `./gradlew assembleCommcareDebug` + Expected: BUILD SUCCESSFUL + +- [ ] **Step 4.9: Commit** + + ```bash + git add app/src/org/commcare/connect/network/ApiEndPoints.java \ + app/src/org/commcare/connect/network/ApiService.java \ + app/src/org/commcare/connect/network/ApiPersonalId.java \ + app/src/org/commcare/connect/network/connectId/PersonalIdApiHandler.java \ + app/src/org/commcare/connect/network/connectId/parser/SendEmailOtpResponseParser.kt \ + app/src/org/commcare/connect/network/connectId/parser/VerifyEmailOtpResponseParser.kt + git commit -m "[AI] Add email OTP API endpoints, service calls, parsers, and handler methods" + ``` + +--- + +## Chunk 3: Email Entry Fragment + +### Task 5: Create email entry layout and fragment + +**Files:** +- Create: `app/res/layout/fragment_personalid_email.xml` +- Create: `app/src/org/commcare/fragments/personalId/PersonalIdEmailFragment.kt` + +- [ ] **Step 5.1: Create fragment_personalid_email.xml** + + Model closely after `app/res/layout/screen_personalid_name.xml`. The layout must include: + - `ScrollView` root with ID `personalid_email_scroll_view` + - `TextInputLayout` + `TextInputEditText` with ID `emailTextValue` (inputType="textEmailAddress") + - Continue button with ID `personalidEmailContinueButton` + - "Skip" text button with ID `personalidEmailSkipButton` (styled as secondary action) + - Error `TextView` with ID `personalidEmailError` (visibility gone by default) + + ```xml + + + + + + + + + + + + + + + + +