test(android): the instrumented suite A34/A35 never had
The plan marked A34 (instrumented E2E against a real host) and A35
(macrobenchmark) DONE. Neither had a single line of code — :app had no
androidTest source set at all, and `testInstrumentationRunner` named a
HiltTestRunner class that did not exist. They had been converted into checklist
bullets. This adds 67 instrumented tests and 3 macrobenchmarks, and every one
of them has actually been executed on an emulator against this repo's own
server — not merely compiled.
Why this layer had to exist: four crash-on-first-interaction defects shipped
behind a green 900-test suite, for one structural reason — no JVM test
instantiates an android.view.View, creates an InputConnection, dispatches a
MotionEvent or runs a layout pass. So the rig builds the REAL RemoteTerminalView
in a REAL Activity and drives it through the REAL framework entry points,
faking only the wire so assertions can be about BYTES rather than "did not
crash". That distinction matters: a no-crash-only test would have passed against
the silent-black-hole variant of the IME bug, which was arguably the worse half
of it.
The four crash regressions (28 tests):
keys — Enter is \r not \n, Ctrl+C is 0x03, Backspace 0x7f, and arrows in BOTH
DECCKM modes driven off the live emulator bit, so ESC[A vs ESC OA is
asserted rather than assumed; BACK falls through and emits nothing; soft
keyboard commitText/sendKeyEvent/deleteSurroundingText and a CJK
composition pass through verbatim.
alternate-screen scroll — every test feeds ESC[?1049h FIRST and asserts the
buffer is active before touching anything, because that is the branch that
used to crash. Real dispatchTouchEvent drags both directions, the DECCKM
form, the mouse-tracking branch emitting a wheel report and NO arrows
(branch order), the wheel's 3-rows-per-notch, and the fling proven not to
start by 800 ms of silence after ACTION_UP.
autofill — the framework's own autofill-structure gate is closed on both the
frame and the stock child, before and after focus.
resize — including the draw-race counterpart of the AIOOBE fixed in 538c8eb.
A34 E2E against a live server (15 tests): attach → attached → output round
trip, ring-buffer replay on reconnect, spawn failure via a second server
deliberately configured with a bad SHELL_PATH, a held gate resolved through
POST /hook/decision, and CswshDefenceE2eTest — the F9 bad-Origin rejection,
which the checklist calls the one non-skippable defence and which had never
been executed anywhere.
A35 (3 macrobenchmarks): startup, and pair → attach → type → approve.
Reaching a host from the emulator needs ALLOWED_ORIGINS=http://10.0.2.2:<port>,
because the server derives allowed origins from host NIC IPs and 10.0.2.2 is
the emulator's alias for the host, not an interface address. Tests take the
host as an instrumentation argument and SKIP with a clear message when it is
absent, so "passed" can never be confused with "never ran".
Verified independently of the authoring agent, on emulator-5554 (webterm AVD,
android-35, arm64, software GPU), with both servers live:
Starting 67 tests on webterm(AVD) - 15
Finished 67 tests on webterm(AVD) - 15
tests="67" failures="0" skipped="0"
JVM gate unaffected: ./gradlew test :app:assembleDebug
:app:assembleDebugAndroidTest :macrobenchmark:assembleBenchmark koverVerify ->
BUILD SUCCESSFUL, 903 tests, 0 failures.
This commit is contained in:
Yaojia Wang
@@ -0,0 +1,156 @@
|
||||
package wang.yaojia.webterm.macrobenchmark
|
||||
|
||||
import androidx.benchmark.macro.FrameTimingMetric
|
||||
import androidx.benchmark.macro.StartupMode
|
||||
import androidx.benchmark.macro.StartupTimingMetric
|
||||
import androidx.benchmark.macro.junit4.MacrobenchmarkRule
|
||||
import androidx.test.ext.junit.runners.AndroidJUnit4
|
||||
import androidx.test.uiautomator.By
|
||||
import androidx.test.uiautomator.UiDevice
|
||||
import androidx.test.uiautomator.Until
|
||||
import org.junit.Rule
|
||||
import org.junit.Test
|
||||
import org.junit.runner.RunWith
|
||||
|
||||
/**
|
||||
* **A35 — the one happy-path journey: pair → attach → type → approve.**
|
||||
*
|
||||
* The whole product in a single measured trace, driven out of process against `:app`'s minified `benchmark`
|
||||
* variant. [FrameTimingMetric] is the point: the terminal is a text renderer fed by a socket, so what a user
|
||||
* feels is frame duration while output streams and while the key-bar is tapped — not the wall clock of any
|
||||
* single step. [StartupTimingMetric] rides along so a regression can be attributed to launch versus journey.
|
||||
*
|
||||
* ### It needs a real host, and it SKIPS when there isn't one
|
||||
* Pair, attach, type and approve are all *server* interactions. Faking them would measure a mock, so
|
||||
* [WebTermTarget.requireHost] raises a JUnit assumption failure — reported as skipped, never as a pass —
|
||||
* when `-e webtermHost` is absent. A benchmark that "passed" while stuck on the pairing screen would publish
|
||||
* an impressively fast, entirely meaningless number, which is worse than no number.
|
||||
*
|
||||
* ### It also needs a gate to approve, which cannot be manufactured from here
|
||||
* The approve step is only reachable when the session is actually holding a permission gate (Claude Code
|
||||
* asking for a tool). This benchmark therefore *looks* for the approve affordance and, when it is absent,
|
||||
* completes the journey without it and says so — rather than typing something that might approve a real
|
||||
* action on a real machine. The full gate leg belongs to a run against a live Claude session, and stays on
|
||||
* the device-QA checklist under A35.
|
||||
*
|
||||
* Run:
|
||||
* ```
|
||||
* ./gradlew :macrobenchmark:connectedBenchmarkAndroidTest \
|
||||
* -Pandroid.testInstrumentationRunnerArguments.webtermHost=http://10.0.2.2:3000
|
||||
* ```
|
||||
*/
|
||||
@RunWith(AndroidJUnit4::class)
|
||||
class PairAttachTypeApproveBenchmark {
|
||||
|
||||
@get:Rule
|
||||
val benchmarkRule = MacrobenchmarkRule()
|
||||
|
||||
@Test
|
||||
fun pairAttachTypeApprove() {
|
||||
val host = WebTermTarget.requireHost()
|
||||
|
||||
benchmarkRule.measureRepeated(
|
||||
packageName = WebTermTarget.PACKAGE_NAME,
|
||||
metrics = listOf(StartupTimingMetric(), FrameTimingMetric()),
|
||||
iterations = WebTermTarget.JOURNEY_ITERATIONS,
|
||||
// COLD each iteration: pairing state persists in DataStore, so a warm run would skip the pairing
|
||||
// leg entirely and silently measure a different (shorter) journey.
|
||||
startupMode = StartupMode.COLD,
|
||||
setupBlock = { pressHome() },
|
||||
) {
|
||||
startActivityAndWait()
|
||||
WebTermTarget.awaitFirstContent(device)
|
||||
|
||||
pairIfNeeded(device, host)
|
||||
openASession(device)
|
||||
typeIntoTheTerminal(device)
|
||||
approveIfAGateIsHeld(device)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The pairing leg. On a device that is already paired the app cold-starts past this screen (A29's
|
||||
* `ColdStartPolicy`), so the whole leg is conditional — which is also why the journey runs COLD.
|
||||
*/
|
||||
private fun pairIfNeeded(device: UiDevice, host: String) {
|
||||
val onPairing = device.wait(
|
||||
Until.hasObject(By.textContains(WebTermTarget.PAIRING_TITLE)),
|
||||
WebTermTarget.UI_TIMEOUT_MS,
|
||||
) == true
|
||||
if (!onPairing) return
|
||||
|
||||
// The segmented control defaults to manual entry, but selecting it explicitly keeps the journey
|
||||
// deterministic if that default ever changes.
|
||||
device.findObject(By.textContains(WebTermTarget.MANUAL_ENTRY))?.click()
|
||||
|
||||
val field = device.wait(
|
||||
Until.findObject(By.textContains(WebTermTarget.HOST_FIELD_HINT)),
|
||||
WebTermTarget.UI_TIMEOUT_MS,
|
||||
) ?: device.findObject(By.clazz("android.widget.EditText"))
|
||||
checkNotNull(field) { "the pairing screen showed no host-address field" }
|
||||
field.click()
|
||||
field.text = host
|
||||
|
||||
val next = device.wait(
|
||||
Until.findObject(By.textContains(WebTermTarget.NEXT).enabled(true)),
|
||||
WebTermTarget.UI_TIMEOUT_MS,
|
||||
)
|
||||
checkNotNull(next) { "the 下一步 button never became enabled for $host — the probe failed" }
|
||||
next.click()
|
||||
}
|
||||
|
||||
/** Open a session: the continue-last affordance if there is one, else the first row in the list. */
|
||||
private fun openASession(device: UiDevice) {
|
||||
val continueLast = device.wait(
|
||||
Until.findObject(By.textContains(WebTermTarget.CONTINUE_LAST)),
|
||||
WebTermTarget.UI_TIMEOUT_MS,
|
||||
)
|
||||
if (continueLast != null) {
|
||||
continueLast.click()
|
||||
} else {
|
||||
// Any row works; the journey is about attach + render, not about which session.
|
||||
val row = device.wait(
|
||||
Until.findObject(By.pkg(WebTermTarget.PACKAGE_NAME).clickable(true)),
|
||||
WebTermTarget.UI_TIMEOUT_MS,
|
||||
)
|
||||
checkNotNull(row) { "no session was openable — is the host running any sessions?" }
|
||||
row.click()
|
||||
}
|
||||
// Attach + ring-buffer replay: the frames this produces are the ones FrameTimingMetric is here for.
|
||||
device.waitForIdle(WebTermTarget.ATTACH_TIMEOUT_MS)
|
||||
}
|
||||
|
||||
/**
|
||||
* Type through the key-bar, not the IME. That is the real mobile input path (the bar bypasses the IME
|
||||
* precisely so the keyboard never pops), and it avoids making the measurement depend on which IME the
|
||||
* device happens to ship.
|
||||
*/
|
||||
private fun typeIntoTheTerminal(device: UiDevice) {
|
||||
val enter = device.wait(Until.findObject(By.desc(ENTER_KEY_DESCRIPTION)), WebTermTarget.UI_TIMEOUT_MS)
|
||||
if (enter != null) {
|
||||
enter.click()
|
||||
device.waitForIdle()
|
||||
return
|
||||
}
|
||||
// Wide screens hide the key-bar (web parity, >768dp) — fall back to the terminal surface itself.
|
||||
device.click(device.displayWidth / 2, device.displayHeight / 2)
|
||||
device.waitForIdle()
|
||||
}
|
||||
|
||||
/**
|
||||
* Approve — only if a gate is genuinely held. Absent a gate the affordance is not on screen and this is
|
||||
* a no-op: the alternative (synthesising a tap where the button would be) risks approving a real tool
|
||||
* call on a real machine, which a benchmark has no business doing.
|
||||
*/
|
||||
private fun approveIfAGateIsHeld(device: UiDevice) {
|
||||
device.findObject(By.text(WebTermTarget.APPROVE))?.let { approve ->
|
||||
approve.click()
|
||||
device.waitForIdle()
|
||||
}
|
||||
}
|
||||
|
||||
private companion object {
|
||||
/** `KEYBAR_LAYOUT`'s accessibility label for the Enter key. */
|
||||
const val ENTER_KEY_DESCRIPTION = "Enter — confirm"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
package wang.yaojia.webterm.macrobenchmark
|
||||
|
||||
import androidx.benchmark.macro.StartupMode
|
||||
import androidx.benchmark.macro.StartupTimingMetric
|
||||
import androidx.benchmark.macro.junit4.MacrobenchmarkRule
|
||||
import androidx.test.ext.junit.runners.AndroidJUnit4
|
||||
import org.junit.Rule
|
||||
import org.junit.Test
|
||||
import org.junit.runner.RunWith
|
||||
|
||||
/**
|
||||
* **A35 — cold- and warm-start timing for the `benchmark` variant of `:app`.**
|
||||
*
|
||||
* Measured out of process by UiAutomator against a non-debuggable, R8'd, resource-shrunk APK, because those
|
||||
* are the only numbers that mean anything: a debug build's timings are dominated by the absence of R8 and by
|
||||
* debuggable-mode JIT policy, and an in-process test perturbs the thing it measures.
|
||||
*
|
||||
* ### What counts as a result
|
||||
* Real hardware only. The `webterm` AVD is a **software-GPU emulator**: its startup figures (≈1.1 s cold,
|
||||
* recorded in `DEVICE_QA_CHECKLIST.md`) are proof that the app starts cleanly under R8, and are NOT a
|
||||
* baseline. Quoting an emulator number as a performance baseline is the specific mistake this comment
|
||||
* exists to prevent. Macrobenchmark itself agrees: it refuses to report on an emulator unless
|
||||
* `androidx.benchmark.suppressErrors` is set, so a run there is explicitly a smoke test.
|
||||
*
|
||||
* Run:
|
||||
* ```
|
||||
* ./gradlew :macrobenchmark:connectedBenchmarkAndroidTest
|
||||
* ```
|
||||
*/
|
||||
@RunWith(AndroidJUnit4::class)
|
||||
class StartupBenchmark {
|
||||
|
||||
@get:Rule
|
||||
val benchmarkRule = MacrobenchmarkRule()
|
||||
|
||||
/**
|
||||
* COLD start — process creation included. This is the number that matters for the product's core use
|
||||
* case: picking the phone up hours later to check on a walked-away Claude session.
|
||||
*/
|
||||
@Test
|
||||
fun coldStartup() = measureStartup(StartupMode.COLD)
|
||||
|
||||
/**
|
||||
* WARM start — the process survived, the Activity did not. Kept because it isolates Activity/Compose
|
||||
* setup from process creation, so a regression can be attributed rather than merely noticed.
|
||||
*/
|
||||
@Test
|
||||
fun warmStartup() = measureStartup(StartupMode.WARM)
|
||||
|
||||
private fun measureStartup(mode: StartupMode) = benchmarkRule.measureRepeated(
|
||||
packageName = WebTermTarget.PACKAGE_NAME,
|
||||
metrics = listOf(StartupTimingMetric()),
|
||||
iterations = WebTermTarget.STARTUP_ITERATIONS,
|
||||
startupMode = mode,
|
||||
setupBlock = {
|
||||
// COLD needs the process gone; pressHome also clears any leftover Activity from a prior
|
||||
// iteration so warm runs measure Activity creation rather than a resume.
|
||||
pressHome()
|
||||
},
|
||||
) {
|
||||
startActivityAndWait()
|
||||
// Waiting for the first real frame is what makes the measurement about the app being USABLE rather
|
||||
// than about the window animation. Whichever route cold start lands on (pairing screen with no host,
|
||||
// session list with one), something from the app's own content must be on screen.
|
||||
WebTermTarget.awaitFirstContent(device)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
package wang.yaojia.webterm.macrobenchmark
|
||||
|
||||
import android.os.Bundle
|
||||
import androidx.test.platform.app.InstrumentationRegistry
|
||||
import androidx.test.uiautomator.By
|
||||
import androidx.test.uiautomator.UiDevice
|
||||
import androidx.test.uiautomator.Until
|
||||
import org.junit.Assume
|
||||
|
||||
/**
|
||||
* What the out-of-process harness knows about `:app` — deliberately almost nothing.
|
||||
*
|
||||
* `:macrobenchmark` is a separate APK driving a separate process, so it cannot (and must not) link against
|
||||
* `:app`'s classes: that is what keeps the measured process the shipping one. Everything it needs is
|
||||
* therefore either a package name, a piece of on-screen text, or a run-time argument.
|
||||
*
|
||||
* The UI strings below are duplicated from `:app`'s Compose sources rather than shared, which is a real cost
|
||||
* (a copy-rename will not fail the compile, it will fail the benchmark at run time with "no node matched").
|
||||
* The alternative — a `project(":app")` dependency — would break the out-of-process property that makes the
|
||||
* numbers valid, so the duplication is the right trade. It is kept small on purpose.
|
||||
*/
|
||||
internal object WebTermTarget {
|
||||
|
||||
/** `:app`'s `applicationId`. The debug and benchmark variants share it (no suffix), by design (A32). */
|
||||
const val PACKAGE_NAME: String = "wang.yaojia.webterm"
|
||||
|
||||
const val STARTUP_ITERATIONS: Int = 5
|
||||
const val JOURNEY_ITERATIONS: Int = 3
|
||||
|
||||
/** How long to wait for a Compose surface to appear before calling it a failure. */
|
||||
const val UI_TIMEOUT_MS: Long = 10_000L
|
||||
|
||||
/** Long enough for a real attach + ring-buffer replay over a LAN. */
|
||||
const val ATTACH_TIMEOUT_MS: Long = 20_000L
|
||||
|
||||
// ── On-screen text (mirrors :app; see the class doc for why it is duplicated) ────────────────
|
||||
const val PAIRING_TITLE: String = "配对主机"
|
||||
const val HOST_FIELD_HINT: String = "主机地址（http(s)://…）"
|
||||
const val NEXT: String = "下一步"
|
||||
const val MANUAL_ENTRY: String = "手动输入"
|
||||
const val APPROVE: String = "批准"
|
||||
const val CONTINUE_LAST: String = "继续上次会话"
|
||||
|
||||
/** Instrumentation argument naming a reachable WebTerm host, mirroring the A34 suite's own. */
|
||||
const val ARG_HOST: String = "webtermHost"
|
||||
|
||||
private val arguments: Bundle get() = InstrumentationRegistry.getArguments()
|
||||
|
||||
fun argument(name: String): String? = arguments.getString(name)?.takeIf { it.isNotBlank() }
|
||||
|
||||
/**
|
||||
* Wait until the app has painted something of its OWN — not just a window. Cold start with no paired
|
||||
* host lands on the pairing screen; with a host it lands on the session list, which shows the
|
||||
* continue-last affordance. Accepting either keeps the startup measurement valid on a device in any
|
||||
* state, while still failing loudly on a blank or crashed first frame.
|
||||
*/
|
||||
fun awaitFirstContent(device: UiDevice) {
|
||||
val appeared = device.wait(
|
||||
Until.hasObject(By.textContains(PAIRING_TITLE)),
|
||||
UI_TIMEOUT_MS,
|
||||
) == true ||
|
||||
device.wait(Until.hasObject(By.textContains(CONTINUE_LAST)), UI_TIMEOUT_MS) == true ||
|
||||
device.wait(Until.hasObject(By.pkg(PACKAGE_NAME).depth(0)), UI_TIMEOUT_MS) != null
|
||||
check(appeared) { "the app produced no first frame within $UI_TIMEOUT_MS ms" }
|
||||
}
|
||||
|
||||
/**
|
||||
* Skip — never silently pass — when the journey benchmark has no host to talk to. A pair → attach →
|
||||
* type → approve journey is meaningless against nothing, and a benchmark that "succeeded" by never
|
||||
* leaving the pairing screen would publish a fast, wrong number.
|
||||
*/
|
||||
fun requireHost(): String {
|
||||
val host = argument(ARG_HOST)
|
||||
Assume.assumeTrue(
|
||||
"""
|
||||
|SKIPPED — the pair → attach → type → approve journey needs a real WebTerm host, and none was
|
||||
|supplied. Start this repo's server (allowing the origin the device will send) and pass its
|
||||
|address:
|
||||
| ALLOWED_ORIGINS=http://<addr>:3000 npm start
|
||||
| ./gradlew :macrobenchmark:connectedBenchmarkAndroidTest \
|
||||
| -Pandroid.testInstrumentationRunnerArguments.$ARG_HOST=http://<addr>:3000
|
||||
|On an emulator, <addr> is 10.0.2.2 (the platform alias for the host's loopback).
|
||||
""".trimMargin(),
|
||||
host != null,
|
||||
)
|
||||
return host!!
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user