Jake Wharton2024-03-28T21:01:40+00:00https://jakewharton.com/Jake WhartonNonsensical Maven is still a Gradle problem2024-03-28T00:00:00+00:00https://jakewharton.com/nonsensical-maven-is-still-a-gradle-problem<p>There was a time when I used Maven heavily, but today all the libraries I work on build with Gradle. Even though I’m publishing with Gradle, consumers can use Gradle, Maven, Bazel, jars in <code class="highlighter-rouge">libs/</code> (but please don’t), or anything else. That’s a huge JVM ecosystem win!</p>
<p>In general, I don’t have to think about what build system someone is using. I’m not here to debate subjective pros and cons of one versus any other. There is one notable exception, however. Maven’s dependency resolution strategy is objectively bonkers. And if we want to support Maven consumers, we need to think about it.</p>
<p>If you already are familiar with the concept of dependency resolution, you can skip to <a href="#the-nonsense">the nonsense</a>.</p>
<h3 id="dependency-resolution-primer">Dependency resolution primer</h3>
<p>Chances are your build system of choice (or a separate dependency resolver tool) gives you a declarative way to describe your dependencies. At build time, those declarations are resolved to <code class="highlighter-rouge">.jar</code>s which can be put on the compiler classpath.</p>
<p>Sometimes we call this a <em>dependency tree</em>, but it’s actually a <em>dependency graph</em>, as separate nodes can converge back to something common to both.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Project (build.gradle)
├── A
│ └── B
│ └── C v1.0
└── D
└── C v1.0
</code></pre></div></div>
<p>If library <code class="highlighter-rouge">B</code> and library <code class="highlighter-rouge">D</code> agree on the version of library <code class="highlighter-rouge">C</code>, then that is the <code class="highlighter-rouge">.jar</code> version which is used. If they disagree on versions, some policy needs to decide the appropriate single version to use.</p>
<p>Pop quiz: If library <code class="highlighter-rouge">B</code> wants version 1.1 of library <code class="highlighter-rouge">C</code>, and library <code class="highlighter-rouge">D</code> wants version 1.0 of library <code class="highlighter-rouge">C</code>, which single version of <code class="highlighter-rouge">C</code> should we use?</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Project (build.gradle)
├── A
│ └── B
│ └── C v1.1
└── D
└── C v1.0
</code></pre></div></div>
<p>This is not a trick question. Hopefully the answer feels obvious: you use the newer version, 1.1. That version is <em>probably</em> compatible with 1.0, so it’s safe for both library <code class="highlighter-rouge">B</code> and library <code class="highlighter-rouge">D</code> to use. We can’t know for sure, to be clear, but it’s a safe choice. This behavior is the default in many dependency resolvers, including the one inside Gradle.</p>
<h3 id="the-nonsense">The nonsense</h3>
<p>When building with Maven, given two dependencies who disagree on a transitive dependency version, the default resolution strategy is… uh… let’s say “interesting”. From <a href="https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Transitive_Dependencies">their docs</a>,</p>
<blockquote>
<p>Maven picks the “nearest definition”. That is, it uses the version of the closest dependency to your project in the tree of dependencies. … Note that if two dependency versions are at the same depth in the dependency tree, the first declaration wins.</p>
</blockquote>
<p>So in a dependency graph, if library <code class="highlighter-rouge">B</code> wants version 1.1 of library <code class="highlighter-rouge">C</code>, and library <code class="highlighter-rouge">D</code> wants version 1.0 of library <code class="highlighter-rouge">C</code>, which single version of <code class="highlighter-rouge">C</code> does Maven choose?</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Project (pom.xml)
├── A
│ └── B
│ └── C v1.1
└── D
└── C v1.0
</code></pre></div></div>
<p>The final build will use version 1.0 of library <code class="highlighter-rouge">C</code>. Wat.</p>
<p>If library <code class="highlighter-rouge">B</code> was using a new API from library <code class="highlighter-rouge">C</code>’s version 1.1, the application will throw a <code class="highlighter-rouge">NoSuchMethodException</code> or the like at runtime.</p>
<p>As if that wasn’t bad enough, disagreements which occur on the same conceptual level of the graph are resolved by whichever comes first. If our project replaces its library <code class="highlighter-rouge">A</code> with direct usage of library <code class="highlighter-rouge">B</code>, suddenly the resolved version is 1.1 because it came first.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Project (pom.xml)
├── B
│ └── C v1.1
└── D
└── C v1.0
</code></pre></div></div>
<p>But if by chance library <code class="highlighter-rouge">D</code> was declared first in the <code class="highlighter-rouge">pom.xml</code> then oops! we’re back to getting 1.0.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Project (pom.xml)
├── D
│ └── C v1.0
└── B
└── C v1.1
</code></pre></div></div>
<p>This behavior is not user-friendly. You can always force a specific version by declaring it directly in your <code class="highlighter-rouge">pom.xml</code>, but that also means you take ownership of monitoring the versions requested by the entire dependency graph and ensuring you declare the one you need. Gee, that sounds like something it should do for you.</p>
<h3 id="still-a-gradle-problem">Still a Gradle problem</h3>
<p>So Maven has some nonsensical dependency resolution semantics. Why should you, a Gradle user, even care?</p>
<p>In the examples above, the version mismatches were demonstrated using peer dependencies on the Maven project. But disagreements can occur within the transitive graph of a single Gradle-built library.</p>
<p>If I am the author of library <code class="highlighter-rouge">A</code> from above, I only have a dependency on library <code class="highlighter-rouge">B</code> and it has a dependency on library <code class="highlighter-rouge">C</code>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Project A (build.gradle)
└── B
└── C v1.1
</code></pre></div></div>
<p>If I want to start using library <code class="highlighter-rouge">C</code>, I <em>may</em> add my own dependency (such as if <code class="highlighter-rouge">C</code> is an implementation dependency of <code class="highlighter-rouge">B</code>) and select an older version.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> Project A (build.gradle)
├── B
│ └── C v1.1
<span class="gi">+└── C v1.0
</span></code></pre></div></div>
<p>I have just unknowingly created a time bomb for all of my Maven consumers.</p>
<h3 id="not-a-hypothetical">Not a hypothetical</h3>
<p>Is this a frequent problem? Seems like no. Is this a real problem? Absolutely.</p>
<p>OkHttp 4.12 <a href="https://repo1.maven.org/maven2/com/squareup/okhttp3/okhttp/4.12.0/okhttp-4.12.0.pom">ships with two dependencies</a>: Okio 3.6 and the Kotlin stdlib 1.8.21. Okio 3.6, however, <a href="https://repo1.maven.org/maven2/com/squareup/okio/okio-jvm/3.6.0/okio-jvm-3.6.0.pom">depends on</a> Kotlin stdlib 1.9.10.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>OkHttp v4.12.0
├── Okio v3.6.0
│ └── Kotlin stdlib v1.9.10
└── Kotlin stdlib v1.8.21
</code></pre></div></div>
<p>This specific configuration is probably okay in practice, as Okio is unlikely to have used anything new. In general, however, the ability to create such a dependency graph with a mismatch is setting our Maven users up for future failure.</p>
<h3 id="detecting-from-maven">Detecting from Maven</h3>
<p>If you are a Maven user, you can eagerly detect this case by using the <a href="https://maven.apache.org/enforcer/maven-enforcer-plugin/index.html">Maven enforcer plugin</a> and its built-in <a href="https://maven.apache.org/enforcer/enforcer-rules/dependencyConvergence.html">dependency convergence rule</a>.</p>
<p>A Maven project with an OkHttp 4.12 dependency will now fail like this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[ERROR] Rule 0: org.apache.maven.enforcer.rules.dependency.DependencyConvergence failed with message:
[ERROR] Failed while enforcing releasability.
[ERROR]
[ERROR] Dependency convergence error for org.jetbrains.kotlin:kotlin-stdlib-jdk8:jar:1.9.10 paths to dependency are:
[ERROR] +-com.example:example:jar:1.0-SNAPSHOT
[ERROR] +-com.squareup.okhttp3:okhttp:jar:4.12.0:compile
[ERROR] +-com.squareup.okio:okio:jar:3.6.0:compile
[ERROR] +-com.squareup.okio:okio-jvm:jar:3.6.0:compile
[ERROR] +-org.jetbrains.kotlin:kotlin-stdlib-jdk8:jar:1.9.10:compile
[ERROR] and
[ERROR] +-com.example:example:jar:1.0-SNAPSHOT
[ERROR] +-com.squareup.okhttp3:okhttp:jar:4.12.0:compile
[ERROR] +-org.jetbrains.kotlin:kotlin-stdlib-jdk8:jar:1.8.21:compile
</code></pre></div></div>
<p>Now a Maven consumer can temporarily resolve the conflict, and go and ask the library maintainer to correct this configuration.</p>
<h3 id="ignoring-the-problem-with-gradle">Ignoring the problem with Gradle</h3>
<p>Since Gradle is going to resolve to the newest version of a dependency, your tests end up running with the newest version rather than the declared version. As such, you can tell Gradle to replace your declared version with the resolved version when publishing.</p>
<p>This behavior is not Gradle’s default, so we must choose it when setting up publishing. The <a href="https://docs.gradle.org/current/userguide/publishing_maven.html#publishing_maven:resolved_dependencies">Gradle docs</a> has an example:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">publishing</span> <span class="o">{</span>
<span class="n">publications</span> <span class="o">{</span>
<span class="n">mavenJava</span><span class="o">(</span><span class="n">MavenPublication</span><span class="o">)</span> <span class="o">{</span>
<span class="n">versionMapping</span> <span class="o">{</span>
<span class="n">usage</span><span class="o">(</span><span class="s1">'java-api'</span><span class="o">)</span> <span class="o">{</span>
<span class="n">fromResolutionOf</span><span class="o">(</span><span class="s1">'runtimeClasspath'</span><span class="o">)</span>
<span class="o">}</span>
<span class="n">usage</span><span class="o">(</span><span class="s1">'java-runtime'</span><span class="o">)</span> <span class="o">{</span>
<span class="n">fromResolutionResult</span><span class="o">()</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>There’s very little harm in doing this, and it will prevent the Maven issue completely. Nice!</p>
<p>The tradeoff is that it somewhat undermines the versions you declare. Keep in mind, though, even if you declare a dependency version and resolve to that same version, a downstream consumer may resolve a newer version or force an older version.</p>
<p>For me, I want the versions which I declare to be those which are resolved, at least local to my project. So this solution isn’t going to work, but it might for your projects.</p>
<p>(Thanks to <a href="https://mastodon.social/@eskatos/112174733070682832">Paul Merlin</a> for suggesting this solution which was added after initial publishing)</p>
<h3 id="trying-to-fix-with-gradle">Trying to fix with Gradle</h3>
<p>I’m going to outright dismiss “just don’t use Maven” as a potential fix. There are lots of reasons not to use Maven that one can explore elsewhere. Ultimately it remains in widespread use, and you can either be sympathetic to those users or not.</p>
<p>Library developers using Gradle could change the default resolution strategy to <a href="https://docs.gradle.org/current/dsl/org.gradle.api.artifacts.ResolutionStrategy.html#org.gradle.api.artifacts.ResolutionStrategy:failOnVersionConflict()">fail on version conflict</a>. This does precisely what it says, fails your build if the transitive graph contains conflicts.</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// OkHttp's build.gradle</span>
<span class="n">dependencies</span> <span class="o">{</span>
<span class="n">implementation</span> <span class="s1">'com.squareup.okio:okio:3.6.0'</span>
<span class="n">implementation</span> <span class="s1">'org.jetbrains.kotlin:kotlin-stdlib:1.8.21'</span>
<span class="o">}</span>
<span class="n">configurations</span><span class="o">.</span><span class="na">configureEach</span> <span class="o">{</span>
<span class="n">resolutionStrategy</span><span class="o">.</span><span class="na">failOnVersionConflict</span><span class="o">()</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Now when building we get a failure:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Execution failed for task ':compileJava'.
> Could not resolve all dependencies for configuration ':compileClasspath'.
> Conflicts found for the following modules:
- org.jetbrains.kotlin:kotlin-stdlib-common between versions 1.9.10 and 1.8.21
- org.jetbrains.kotlin:kotlin-stdlib between versions 1.9.10 and 1.8.21
</code></pre></div></div>
<p>The failure suggests running <code class="highlighter-rouge">dependencyInsight</code>, which shows you a wall of text containing the subgraph of affected dependencies which led to the conflict.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>> Task :dependencyInsight
Dependency resolution failed because of conflicts on the following modules:
- org.jetbrains.kotlin:kotlin-stdlib-common between versions 1.9.10 and 1.8.21
org.jetbrains.kotlin:kotlin-stdlib-common:1.9.10
Variant compile:
| Attribute Name | Provided | Requested |
|--------------------------------|----------|--------------|
| org.gradle.status | release | |
| org.gradle.category | library | library |
| org.gradle.libraryelements | jar | classes |
| org.gradle.usage | java-api | java-api |
| org.gradle.dependency.bundling | | external |
| org.gradle.jvm.environment | | standard-jvm |
| org.gradle.jvm.version | | 21 |
Selection reasons:
- By conflict resolution: between versions 1.9.10 and 1.8.21
org.jetbrains.kotlin:kotlin-stdlib-common:1.9.10
+--- com.squareup.okio:okio-jvm:3.6.0
| \--- com.squareup.okio:okio:3.6.0
| \--- compileClasspath
\--- org.jetbrains.kotlin:kotlin-stdlib:1.9.10
+--- compileClasspath (requested org.jetbrains.kotlin:kotlin-stdlib:1.8.21)
+--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.9.10
| \--- com.squareup.okio:okio-jvm:3.6.0 (*)
\--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.9.10
\--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.9.10 (*)
</code></pre></div></div>
<p>The fix for OkHttp is simple: upgrade to a matching version.</p>
<p>Unfortunately, if you upgrade to a version that’s newer than your transitive dependency, the build still fails.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>> Could not resolve all dependencies for configuration ':compileClasspath'.
> Conflicts found for the following modules:
- org.jetbrains.kotlin:kotlin-stdlib between versions 1.9.23 and 1.9.10
- org.jetbrains.kotlin:kotlin-stdlib-jdk8 between versions 1.9.10 and 1.8.0
- org.jetbrains.kotlin:kotlin-stdlib-common between versions 1.9.23 and 1.9.10
- org.jetbrains.kotlin:kotlin-stdlib-jdk7 between versions 1.9.10 and 1.8.0
</code></pre></div></div>
<p>You have to force the use of 1.9.23 everywhere, but doing so will ironically prevent <code class="highlighter-rouge">failOnVersionConflict()</code> from detecting mismatches in the future.</p>
<p>Gradle has other mechanisms like <a href="https://docs.gradle.org/current/userguide/rich_versions.html">constraints</a> and <a href="https://docs.gradle.org/current/userguide/resolution_rules.html">resolution strategy callbacks</a> that have tons of power to customize dependency resolution, but none provide the ability to reject upgrades. I would love to be corrected on this, but I spent a few days searching and experimenting with no success. Instead, we have to build our own solution.</p>
<h3 id="actually-fixing-with-gradle">Actually fixing with Gradle</h3>
<p>I wrote a task which consumes the dependency graph and checks if the first-order dependencies (i.e., those your project declared directly) select the same version as they request.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>> Task :sympathyForMrMaven FAILED
e: org.jetbrains.kotlin:kotlin-stdlib:1.8.21 changed to 1.9.10
* What went wrong:
Execution failed for task ':sympathyForMrMaven'.
> Declared dependencies were upgraded transitively. See task output above. Please update their versions.
</code></pre></div></div>
<p>When I bump my declaration to 1.9.10 to match, or even 1.9.23 which is the latest right now, the task no longer fails.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>BUILD SUCCESSFUL in 354ms
4 actionable tasks: 4 executed
</code></pre></div></div>
<p>This is what I hacked up in Groovy <em>very</em> quickly this morning (and to finish the damn post):</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">def</span> <span class="n">fail</span> <span class="o">=</span> <span class="kc">false</span>
<span class="kt">def</span> <span class="n">root</span> <span class="o">=</span> <span class="n">configuration</span><span class="o">.</span><span class="na">incoming</span><span class="o">.</span><span class="na">resolutionResult</span><span class="o">.</span><span class="na">rootComponent</span><span class="o">.</span><span class="na">get</span><span class="o">()</span>
<span class="o">((</span><span class="n">ResolvedComponentResult</span><span class="o">)</span> <span class="n">root</span><span class="o">).</span><span class="na">dependencies</span><span class="o">.</span><span class="na">forEach</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span><span class="n">it</span> <span class="k">instanceof</span> <span class="n">ResolvedDependencyResult</span><span class="o">)</span> <span class="o">{</span>
<span class="kt">def</span> <span class="n">rdr</span> <span class="o">=</span> <span class="n">it</span> <span class="k">as</span> <span class="n">ResolvedDependencyResult</span>
<span class="kt">def</span> <span class="n">requested</span> <span class="o">=</span> <span class="n">rdr</span><span class="o">.</span><span class="na">requested</span>
<span class="kt">def</span> <span class="n">selected</span> <span class="o">=</span> <span class="n">rdr</span><span class="o">.</span><span class="na">selected</span>
<span class="k">if</span> <span class="o">(</span><span class="n">requested</span> <span class="k">instanceof</span> <span class="n">ModuleComponentSelector</span><span class="o">)</span> <span class="o">{</span>
<span class="kt">def</span> <span class="n">requestedVersion</span> <span class="o">=</span> <span class="o">(</span><span class="n">requested</span> <span class="k">as</span> <span class="n">ModuleComponentSelector</span><span class="o">).</span><span class="na">version</span>
<span class="kt">def</span> <span class="n">selectedVersion</span> <span class="o">=</span> <span class="n">selected</span><span class="o">.</span><span class="na">moduleVersion</span><span class="o">.</span><span class="na">version</span>
<span class="k">if</span> <span class="o">(</span><span class="n">requestedVersion</span> <span class="o">!=</span> <span class="n">selectedVersion</span><span class="o">)</span> <span class="o">{</span>
<span class="n">logger</span><span class="o">.</span><span class="na">log</span><span class="o">(</span><span class="n">ERROR</span><span class="o">,</span> <span class="s2">"e: ${rdr.requested} changed to ${selectedVersion}"</span><span class="o">)</span>
<span class="n">fail</span> <span class="o">=</span> <span class="kc">true</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="k">if</span> <span class="o">(</span><span class="n">fail</span><span class="o">)</span> <span class="o">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">IllegalStateException</span><span class="o">(</span><span class="s2">"Declared dependencies were upgraded transitively. See task output above. Please update their versions."</span><span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>This needs cleaned up before it can be used generally–sorry! In a long post about how Maven’s dependency resolution is annoying, I instead became <a href="https://jakewharton.com/@jw/112171457869714385"><em>very</em> annoyed at Gradle</a> and just want to stop working on this.</p>
<p>Someone please change it to Java, wrap it in a task, wrap that in a <code class="highlighter-rouge">com.yourname.maven-sympathy</code> plugin, publish to Maven Central, and ping me to update this post. I have about 30 projects I’d love to slap it on, and hopefully other sympathetic library authors who read this post will too!</p>
Gradle toolchains are rarely a good idea2024-03-21T00:00:00+00:00https://jakewharton.com/gradle-toolchains-are-rarely-a-good-idea<p>The <a href="/kotlins-jdk-release-compatibility-flag/">last post</a> featured some Kotlin code inadvertently targeting a new Java API when the build JDK was bumped to 21. This can be solved with the <code class="highlighter-rouge">-Xjdk-release</code> Kotlin compiler flag, or by using Gradle toolchains to build with an old JDK.</p>
<p>If you read the <a href="https://docs.gradle.org/current/userguide/building_java_projects.html#sec:java_cross_compilation">Gradle docs</a>…</p>
<blockquote>
<p>Using Java toolchains is a preferred way to target a language version</p>
</blockquote>
<p>…or the <a href="https://developer.android.com/build/jdks#toolchain">Android docs</a>…</p>
<blockquote>
<p>We recommend that you always specify the Java toolchain</p>
</blockquote>
<p>…you wouldn’t be blamed for thinking Java toolchains are the way to go!</p>
<p>However, Java toolchains are rarely a good idea. Let’s look at why.</p>
<h3 id="bad-docs">Bad docs</h3>
<p>Last week I released a new version of <a href="https://github.com/square/retrofit">Retrofit</a> which uses a Java toolchain to target Java 8. Its use of toolchains was contributed a while ago, and I simply forgot to remove it. As a consequence, <a href="https://square.github.io/retrofit/2.x/retrofit/">its Javadoc</a> was built using JDK 8 and is thus not searchable. Searchable Javadoc came in <a href="https://openjdk.org/jeps/225">JEP 225</a> with JDK 9.</p>
<p>The next release of Retrofit will be made without a toolchain and with the latest JDK. Its docs will have all the Javadoc advancements from the last 10 years including search and better modern HTML/CSS.</p>
<h3 id="resource-ignorance">Resource ignorance</h3>
<p>Old JVMs were somewhat notorious for being ignorant to resource limitations imposed by the system. The rise of containers, especially on CI systems, means your process resource limits are different from those of the host OS. JDK 10 kicked things into high gear with <a href="https://bugs.openjdk.org/browse/JDK-8146115">cgroups support</a> and JDK 15 <a href="https://bugs.openjdk.org/browse/JDK-8230305">extended that to cgroups2</a>.</p>
<p>Both of those changes were backported to the 8 and 11 branches, but since Gradle toolchains will use an already-installed JDK if available you have to have kept your JDK 8 and/or JDK 11 up-to-date. Have you?</p>
<p>Not to stray too far off-topic, but if you installed it with SDKMAN! or similar JDK management tools there’s a good chance it’s wildly out of date. I keep all my JDKs up-to-date by installing them through <a href="https://github.com/mdogan/homebrew-zulu">a Homebrew tap</a> which itself updates automatically using the Azul Zulu API. As long as I do a <code class="highlighter-rouge">brew upgrade</code> every so often, each major JDK release that I have installed will be updated.</p>
<p>Without a Java toolchain, a modern JDK (even an outdated patch release of one) will honor resource limits and perform much better in containerized environments.</p>
<h3 id="compiler-bugs">Compiler bugs</h3>
<p>All software has bugs, and sometimes the JVM, the Java compiler, or both have bugs. When you are using a 10-year-old version of the JVM and Java compiler, you run a much greater risk of compiler bugs, especially around features introduced near to that release.</p>
<p>There were many compilation problems around lambdas which were introduced in Java 8. If you are using the Java compiler from JDK 8 to target Java 8 JVMs you can still run into those bugs. Even if you are keeping your JDK 8 up-to-date many fixes are not backported. You <a href="https://bugs.openjdk.org/browse/JDK-8182401">can find ones</a> on the issue tracker without much effort.</p>
<p>Now is the Java compiler in JDK 22 completely bug-free? No. But is using the Java compiler from JDK 22 on sources targeting Java 8 using only Java 8 language features much safer than using one from JDK 8? Absolutely.</p>
<h3 id="worse-performance">Worse performance</h3>
<p>Oracle and other large JVM shops devote lots of person-hours to making the JVM faster. We have newer garbage collectors that use less memory and consume less CPU. Work that happened on startup gets deferred to first-use to try and spread the cost out over the lifetime of the process. Algorithms and in-memory representations are specialized for common cases.</p>
<p>A language compiler is basically a worst-case scenario for the JVM. Endless string manipulation, object creation, and so so many maps. These areas receive many improvements over the years. My favorite of which is that strings which are ASCII-based suddenly occupy half as much memory in Java 9 than in Java 8. You know what’s often entirely ASCII? Java and Kotlin source code!</p>
<h3 id="not-needed-for-cross-compilation">Not needed for cross-compilation</h3>
<p>Using the Java compiler from JDK 8 I can set <code class="highlighter-rouge">-source</code> and <code class="highlighter-rouge">-target</code> to “1.7” to compile a class that works on a Java 7 JVM. This does not prevent me from using Java 8 APIs, however. You have to add <code class="highlighter-rouge">-bootclasspath</code> with a pointer to a JDK 7 runtime (<code class="highlighter-rouge">rt.jar</code>) so that the compiler knows what APIs are available in Java 7. You could alternatively use a tool like <a href="https://www.mojohaus.org/animal-sniffer/">Animal Sniffer</a> to validate that no APIs newer than Java 7 were used. In this world, just compiling with JDK 7 to target Java 7 might actually just be easier.</p>
<p>In JDK 9, however, this all changed. The compiler now contains a record of all public APIs from every Java version going back to Java 8. It also allows specifying a single compiler flag, <code class="highlighter-rouge">--release</code>, which sets the source code language version, the target bytecode version, and the available runtime APIs to the specified release. There is simply no value in compiling with an older JDK to target an older JVM anymore.</p>
<h3 id="wasted-disk-space">Wasted disk space</h3>
<p>All those JDKs needlessly take up space in your home directory. Each JDK is a few hundred MiB. By default, Gradle will try to match an existing JDK when a toolchain is requested. Project owners can specify additional attributes such as the JDK vendor which might cause existing JDKs to not match. This means even though one project forced you to install Eclipse Temurin JDK 8, another might force Azul Zulu JDK 8. So not only do you now have a bunch of old JDKs, you have two or three copies of each. My JDK cache in <code class="highlighter-rouge">~/.gradle</code> is nearly 2 GiB.</p>
<h3 id="not-the-gradle-jvm">Not the Gradle JVM</h3>
<p>Toolchains are only used for tasks that create a new JVM. That means compilation (of Java or Kotlin) and running unit tests. They do not control the JVM that is used for running the actual Gradle build or any of the plugins therein. If you have minimum requirements there, or in other JVM-based tools which are invoked by the Gradle build, the toolchain does not help you.</p>
<p>If your build already has a minimum JDK requirement then why force installation of old JDKs given the newer one is already available on disk, can cross-compile perfectly, has fewer compiler bugs, builds faster, and respects system CPU and memory limits more effectively?</p>
<h3 id="not-all-bad">Not all bad</h3>
<p>I want to stress that toolchains are unequivocally not a good idea <em>for compilation</em>. They still have utility elsewhere, however.</p>
<p>Retrofit has runtime behavior that changes based on the JVM version on which it’s running. (This is because until Java 16 it took various different hacks to support invoking default methods through a <a href="https://docs.oracle.com/en%2Fjava%2Fjavase%2F22%2Fdocs%2Fapi%2F%2F/java.base/java/lang/reflect/Proxy.html"><code class="highlighter-rouge">Proxy</code></a>.) That code needs to be tested on different JVM versions. As a result, we <a href="/build-on-latest-java-test-through-lowest-java/">compile with the latest Java, but test through the lowest-supported Java</a> using toolchains on the <code class="highlighter-rouge">Test</code> task. No need to worry about the user having weird old JDKs for Java 14 because it’s now installed on-demand when the full test suite is run.</p>
<p>Some tools that dip into JDK internals regularly break on newer versions of the compiler because they rely on unstable APIs. I’m thinking about things like <a href="https://github.com/google/google-java-format">Google Java Format</a> or <a href="https://errorprone.info/">Error-Prone</a>. No need to hold the rest of your project from enjoying the latest JDK, if those tools are run via a <code class="highlighter-rouge">JavaExec</code> task you can use a toolchain to keep them on an older JDK until a newer version is available.</p>
<h3 id="what-do-i-do">What do I do?</h3>
<p>Use the <code class="highlighter-rouge">--release</code> flag if you’re compiling Java! Gradle <a href="https://docs.gradle.org/current/dsl/org.gradle.api.tasks.compile.CompileOptions.html#org.gradle.api.tasks.compile.CompileOptions:release">exposes a property</a> for it now.</p>
<p>Use the <code class="highlighter-rouge">-Xjdk-release</code> flag if you’re compiling Kotlin. Future versions of the Kotlin Gradle plugin will expose a nice DSL property for it.</p>
<p>If you’re <a href="https://developer.android.com/build/jdks#target-compat">targeting Android</a> (with Java, Kotlin, or both) you need only specify the <code class="highlighter-rouge">sourceCompatibility</code> (for Java) and <code class="highlighter-rouge">jvmTarget</code> (for Kotlin). You don’t need the <code class="highlighter-rouge">targetCompatibility</code> as it will default to match the <code class="highlighter-rouge">sourceCompatibility</code>.</p>
<p>No matter what the Gradle or Android docs tell you, don’t use a toolchain! Save toolchains for JVM unit tests or incompatible tools.</p>
Kotlin's JDK release compatibility flag2024-03-13T00:00:00+00:00https://jakewharton.com/kotlins-jdk-release-compatibility-flag<p>Yesterday, our Android app crashed with a weird <code class="highlighter-rouge">NoSuchMethodError</code>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>java.lang.NoSuchMethodError: No interface method removeFirst()Ljava/lang/Object; in class Ljava/util/List; or its super classes (declaration of 'java.util.List' appears in /apex/com.android.art/javalib/core-oj.jar)
at app.cash.redwood.lazylayout.widget.LazyListUpdateProcessor.onEndChanges(SourceFile:165)
at app.cash.redwood.lazylayout.view.ViewLazyList.onEndChanges(SourceFile:210)
at app.cash.redwood.protocol.widget.ProtocolBridge.sendChanges(SourceFile:125)
at app.cash.redwood.treehouse.ViewContentCodeBinding.receiveChangesOnUiDispatcher(SourceFile:419)
at app.cash.redwood.treehouse.ViewContentCodeBinding$sendChanges$1.invokeSuspend(SourceFile:383)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(SourceFile:33)
at kotlinx.coroutines.DispatchedTask.run(SourceFile:104)
at android.os.Handler.handleCallback(Handler.java:938)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loop(Looper.java:250)
at android.app.ActivityThread.main(ActivityThread.java:7868)
</code></pre></div></div>
<p><a href="https://github.com/cashapp/redwood/blob/2db41653e3887387b8c8468cb3f01d0c326eb39d/redwood-lazylayout-widget/src/commonMain/kotlin/app/cash/redwood/lazylayout/widget/LazyListUpdateProcessor.kt#L165">The offending code</a> is written in Kotlin, and looks like this:</p>
<p><img src="/static/post-image/removeFirst.png" alt="val widget = edit.widgets.removeFirst()" /></p>
<p>The IDE showing an italicized blue style for <code class="highlighter-rouge">removeFirst</code> means <a href="https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.collections/remove-first.html">it’s a Kotlin extension function</a> which compiles down to a static helper in the bytecode.
However, the exception clearly indicates we are calling a member function on <code class="highlighter-rouge">List</code> directly. What gives?</p>
<p>In JDK 21, as part of the <a href="https://openjdk.org/jeps/431">sequenced collection</a> effort, the <code class="highlighter-rouge">List</code> interface <a href="https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/util/List.html#removeFirst()">added <code class="highlighter-rouge">removeFirst()</code> and <code class="highlighter-rouge">removeLast()</code></a> methods. According to the <a href="https://kotlinlang.org/docs/extensions.html#extensions-are-resolved-statically">Kotlin docs on extension functions</a>:</p>
<blockquote>
<p>If a class has a member function, and an extension function is defined which has the same receiver type, the same name, and is applicable to given arguments, the <strong>member always wins</strong>.</p>
</blockquote>
<p>When we bumped our build JDK to 21, the new member became available and accidentally took precedence. Oops!</p>
<p>But wait, we set our Kotlin <code class="highlighter-rouge">jvmTarget</code> to 1.8 in order to be backwards compatible. Is that not enough?</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">val</span> <span class="py">javaVersion</span> <span class="p">=</span> <span class="nc">JavaVersion</span><span class="p">.</span><span class="nc">VERSION_1_8</span>
<span class="n">tasks</span><span class="p">.</span><span class="nf">withType</span><span class="p">(</span><span class="nc">KotlinJvmCompile</span><span class="o">::</span><span class="k">class</span><span class="p">.</span><span class="n">java</span><span class="p">).</span><span class="nf">configureEach</span> <span class="p">{</span>
<span class="n">it</span><span class="p">.</span><span class="n">kotlinOptions</span><span class="p">.</span><span class="n">jvmTarget</span> <span class="p">=</span> <span class="n">javaVersion</span><span class="p">.</span><span class="nf">toString</span><span class="p">()</span>
<span class="p">}</span>
<span class="c1">// Kotlin requires the Java compatibility matches despite have no sources.</span>
<span class="n">tasks</span><span class="p">.</span><span class="nf">withType</span><span class="p">(</span><span class="nc">JavaCompile</span><span class="o">::</span><span class="k">class</span><span class="p">.</span><span class="n">java</span><span class="p">).</span><span class="nf">configureEach</span> <span class="p">{</span>
<span class="n">it</span><span class="p">.</span><span class="n">sourceCompatibility</span> <span class="p">=</span> <span class="n">javaVersion</span><span class="p">.</span><span class="nf">toString</span><span class="p">()</span>
<span class="n">it</span><span class="p">.</span><span class="n">targetCompatibility</span> <span class="p">=</span> <span class="n">javaVersion</span><span class="p">.</span><span class="nf">toString</span><span class="p">()</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This setting controls the Java bytecode version that the Kotlin compiler emits for JVM and Android targets.
We can confirm this is being honored by inspecting the offending class with <code class="highlighter-rouge">javap</code>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ javap -v redwood-lazylayout-widget/build/classes/kotlin/jvm/main/app/cash/redwood/lazylayout/widget/LazyListUpdateProcessor.class | head -8
Classfile redwood-lazylayout-widget/build/classes/kotlin/jvm/main/app/cash/redwood/lazylayout/widget/LazyListUpdateProcessor.class
Last modified Mar 13, 2024; size 16001 bytes
SHA-256 checksum dbeed7bba16c023a98fa356bab7cada7abe686d5da7d4824781790de577e94a2
Compiled from "LazyListUpdateProcessor.kt"
public abstract class app.cash.redwood.lazylayout.widget.LazyListUpdateProcessor<V extends java.lang.Object, W extends java.lang.Object> extends java.lang.Object
minor version: 0
major version: 52
flags: (0x0421) ACC_PUBLIC, ACC_SUPER, ACC_ABSTRACT
</code></pre></div></div>
<p>The classfile’s major version is listed at 52, which we can reverse lookup using <a href="https://javaalmanac.io/bytecode/versions/">a version table</a> and see that this corresponds to Java 8. So we know that’s working, at least.</p>
<p>Further down the output, however, the offending reference can also be seen.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>405: checkcast #101 // class app/cash/redwood/lazylayout/widget/LazyListUpdateProcessor$Edit$Insert
408: invokevirtual #107 // Method app/cash/redwood/lazylayout/widget/LazyListUpdateProcessor$Edit$Insert.getWidgets:()Ljava/util/List;
411: invokeinterface #151, 1 // InterfaceMethod java/util/List.removeFirst:()Ljava/lang/Object;
416: checkcast #121 // class app/cash/redwood/widget/Widget
419: astore 6
</code></pre></div></div>
<p>The reason this can happen is that the Java bytecode version is independent from the set of JDK APIs that you can reference.
This is not unique to Kotlin.
<code class="highlighter-rouge">javac</code>’s <code class="highlighter-rouge">-target</code> flag behaves the same way, as you can see <a href="https://java.godbolt.org/z/rKWv4K9jG">in this Godbolt sample</a>.</p>
<p>This can be fixed with <code class="highlighter-rouge">javac</code> by specifying the <code class="highlighter-rouge">-bootclasspath</code> argument and pointing at the <code class="highlighter-rouge">rt.jar</code> from a JDK 8 install.
The JDK 21 compiler emits a warning telling us to do this when target <em>any</em> bytecode version other than the default:</p>
<blockquote>
<p>warning: [options] bootstrap class path not set in conjunction with -source 8</p>
</blockquote>
<p>Starting with Java 9, <code class="highlighter-rouge">javac</code> has a new flag, <code class="highlighter-rouge">--release</code>, which sets the <code class="highlighter-rouge">-source</code>, <code class="highlighter-rouge">-target</code>, and <code class="highlighter-rouge">-bootclasspath</code> flags automatically to the same version (and doesn’t require having the old JDK available).
If we switch the Java sample to use <code class="highlighter-rouge">--release</code> <a href="https://java.godbolt.org/z/bP6baz9GT">it now fails to compile</a>!</p>
<p>Kotlin 1.7 brought a new flag to <code class="highlighter-rouge">kotlinc</code> (Kotlin’s JVM compiler) which acts just like <code class="highlighter-rouge">javac</code>’s <code class="highlighter-rouge">--release</code>: <code class="highlighter-rouge">-Xjdk-release</code>.
As far as I can tell, this has flown massively under the radar but is an essential piece to the cross-compilation toolkit.</p>
<p>Let’s configure our JVM target’s compilation to use this flag and see what changes.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">kotlin</span><span class="p">.</span><span class="n">targets</span><span class="p">.</span><span class="nf">withType</span><span class="p">(</span><span class="nc">KotlinJvmTarget</span><span class="o">::</span><span class="k">class</span><span class="p">.</span><span class="n">java</span><span class="p">)</span> <span class="p">{</span> <span class="n">target</span> <span class="p">-></span>
<span class="n">target</span><span class="p">.</span><span class="n">compilations</span><span class="p">.</span><span class="nf">configureEach</span> <span class="p">{</span>
<span class="n">it</span><span class="p">.</span><span class="n">kotlinOptions</span><span class="p">.</span><span class="n">freeCompilerArgs</span> <span class="p">+=</span> <span class="nf">listOf</span><span class="p">(</span>
<span class="s">"-Xjdk-release=$javaVersion"</span><span class="p">,</span>
<span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>After compiling and dumping the Java bytecode there is a welcome change.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> 405: checkcast #101 // class app/cash/redwood/lazylayout/widget/LazyListUpdateProcessor$Edit$Insert
408: invokevirtual #107 // Method app/cash/redwood/lazylayout/widget/LazyListUpdateProcessor$Edit$Insert.getWidgets:()Ljava/util/List;
<span class="gd">-411: invokeinterface #151, 1 // InterfaceMethod java/util/List.removeFirst:()Ljava/lang/Object;
</span><span class="gi">+411: invokestatic #152 // Method kotlin/collections/CollectionsKt.removeFirst:(Ljava/util/List;)Ljava/lang/Object;
</span> 414: checkcast #121 // class app/cash/redwood/widget/Widget
417: astore 6
</code></pre></div></div>
<p>With the JDK API unavailable, the <code class="highlighter-rouge">removeFirst</code> extension now resolves to the static method in the Kotlin standard library.</p>
<p>The <code class="highlighter-rouge">-Xjdk-release</code> flag is useful for the Kotlin JVM plugin or the JVM targets of the Kotlin multiplatform plugin to ensure compatibility with your target minimum JVM. Users of the Kotlin Android plugin or the Android targets of the Kotlin multiplatform plugin do not need to do this, as the use of the <code class="highlighter-rouge">android.jar</code> as the boot classpath limits the <code class="highlighter-rouge">java.*</code> APIs to those of your <code class="highlighter-rouge">compileSdk</code> (and Android Lint ensures you don’t use anything newer than your <code class="highlighter-rouge">minSdk</code>).</p>
<p>Unforunately there’s no Gradle DSL for this yet, but <a href="https://youtrack.jetbrains.com/issue/KT-49746/Support-Xjdk-release-in-gradle-toolchain#focus=Comments-27-8935065.0-0">KT-49746</a> tracks that.</p>
<p>If you use Gradle toolchains you don’t have this problem. This is because you actually use the ancient JDK and JVM of your minimum target to run <code class="highlighter-rouge">javac</code> and <code class="highlighter-rouge">kotlinc</code> and miss out on a decade’s worth of compiler improvements. Gradle toolchains are rarely a good idea. But that’s a topic for next week…</p>
Perils of duplicate finding2024-02-14T00:00:00+00:00https://jakewharton.com/perils-of-duplicate-finding<p>Given an array of integers (<code class="highlighter-rouge">[1, 2, 3, 1, 3, 1]</code>), find the elements which are duplicated.
No, we’re not interviewing.
I’m trying to prevent a user from specifying a reserved value twice.</p>
<p>Elsewhere in the file I already have duplicate detection for object tags.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">val</span> <span class="py">dupes</span><span class="p">:</span> <span class="nc">Map</span><span class="p"><</span><span class="nc">Int</span><span class="p">,</span> <span class="nc">List</span><span class="p"><</span><span class="nc">Widget</span><span class="p">>></span> <span class="p">=</span>
<span class="n">widgets</span><span class="p">.</span><span class="nf">groupBy</span><span class="p">(</span><span class="nc">Widget</span><span class="o">::</span><span class="n">tag</span><span class="p">)</span>
<span class="p">.</span><span class="nf">filterValues</span> <span class="p">{</span> <span class="n">it</span><span class="p">.</span><span class="n">size</span> <span class="p">></span> <span class="mi">1</span> <span class="p">}</span>
</code></pre></div></div>
<p>I can do the same technique for the integer array with an identity function and grabbing the resulting keys.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">val</span> <span class="py">dupes</span><span class="p">:</span> <span class="nc">Set</span><span class="p"><</span><span class="nc">Int</span><span class="p">></span> <span class="p">=</span>
<span class="n">ints</span><span class="p">.</span><span class="nf">groupBy</span> <span class="p">{</span> <span class="n">it</span> <span class="p">}</span>
<span class="p">.</span><span class="nf">filterValues</span> <span class="p">{</span> <span class="n">it</span><span class="p">.</span><span class="n">size</span> <span class="p">></span> <span class="mi">1</span> <span class="p">}</span>
<span class="p">.</span><span class="n">keys</span>
</code></pre></div></div>
<p>This prints <code class="highlighter-rouge">[1, 3]</code>.</p>
<p>So… done? Yes!
But no, using the map seems wasteful, right?</p>
<h3 id="attempt-1">Attempt 1</h3>
<p>My first attempt to avoid the map was to remove the set of integers from a list of them.
This should result in a list of any duplicated elements.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">val</span> <span class="py">dupes</span><span class="p">:</span> <span class="nc">List</span><span class="p"><</span><span class="nc">Int</span><span class="p">></span> <span class="p">=</span>
<span class="n">ints</span><span class="p">.</span><span class="nf">toList</span><span class="p">()</span> <span class="p">-</span> <span class="n">ints</span><span class="p">.</span><span class="nf">toSet</span><span class="p">()</span>
</code></pre></div></div>
<p>No matter the content of <code class="highlighter-rouge">ints</code>, this will always print <code class="highlighter-rouge">[]</code>.
Why?</p>
<p>The <a href="https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.collections/minus.html"><code class="highlighter-rouge">minus</code> operator</a> says that it “returns a list containing all elements of the original collection except the elements contained in the given <code class="highlighter-rouge">elements</code> collection”.
So it removes <em>all</em> occurrences of each element in the set from the list.</p>
<p>This is some surprising behavior to hide behind an operator whose signature operates on an <code class="highlighter-rouge">Iterable</code> receiver and <code class="highlighter-rouge">Collection</code> argument.</p>
<h3 id="attempt-2">Attempt 2</h3>
<p>Second attempt switches to <code class="highlighter-rouge">MutableList.removeAll</code> which takes a collection of elements.
The <code class="highlighter-rouge">MutableList.remove</code> function only removes the first occurrence of an element, so this should remove the first occurrence of each element in the set.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">val</span> <span class="py">dupes</span><span class="p">:</span> <span class="nc">List</span><span class="p"><</span><span class="nc">Int</span><span class="p">></span> <span class="p">=</span>
<span class="n">ints</span><span class="p">.</span><span class="nf">toMutableList</span><span class="p">()</span>
<span class="p">.</span><span class="nf">apply</span> <span class="p">{</span> <span class="nf">removeAll</span><span class="p">(</span><span class="n">ints</span><span class="p">.</span><span class="nf">toSet</span><span class="p">())</span> <span class="p">}</span>
</code></pre></div></div>
<p>This once again prints <code class="highlighter-rouge">[]</code>.
But why?</p>
<p><a href="/static/post-image/kotlin-remove-removeAll.png"><img src="/static/post-image/kotlin-remove-removeAll.png" alt="
Screenshot of Kotlin documentation showing two functions:
remove: Removes a single instance of the specified element from this collection, if it is present.
removeAll: Removes all of this collection's elements that are also contained in the specified collection.
" /></a></p>
<p>Kotlin made me a liar.
<code class="highlighter-rouge">MutableList.remove</code> does indeed only remove the first occurrence of the element.
<code class="highlighter-rouge">MutableList.removeAll</code>, however, removes <em>all</em> occurrences of each element in the supplied collection.
That’s quite the subtle asymmetry.</p>
<p>There is no function for removing all occurrences of a single element.
Nor a function to remove only the first occurrences of each element of a supplied collection.</p>
<p>You needn’t be mad at Kotlin, though.
It <a href="https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/util/Collection.html#removeAll(java.util.Collection)">inherited this behavior</a> from Java.</p>
<h3 id="attempt-3">Attempt 3</h3>
<p>Third attempt now with <code class="highlighter-rouge">MutableList.remove</code>.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">val</span> <span class="py">dupes</span><span class="p">:</span> <span class="nc">List</span><span class="p"><</span><span class="nc">Int</span><span class="p">></span> <span class="p">=</span>
<span class="n">ints</span><span class="p">.</span><span class="nf">toMutableList</span><span class="p">()</span>
<span class="p">.</span><span class="nf">apply</span> <span class="p">{</span> <span class="n">ints</span><span class="p">.</span><span class="nf">toSet</span><span class="p">().</span><span class="nf">forEach</span><span class="p">(</span><span class="o">::</span><span class="n">remove</span><span class="p">)</span> <span class="p">}</span>
</code></pre></div></div>
<p>This (finally) prints <code class="highlighter-rouge">[1, 3, 1]</code>.
If we want just the set of duplicates to match the map-based approach above we can tag on a <code class="highlighter-rouge">toSet()</code>.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">val</span> <span class="py">dupes</span><span class="p">:</span> <span class="nc">Set</span><span class="p"><</span><span class="nc">Int</span><span class="p">></span> <span class="p">=</span>
<span class="n">ints</span><span class="p">.</span><span class="nf">toMutableList</span><span class="p">()</span>
<span class="p">.</span><span class="nf">apply</span> <span class="p">{</span> <span class="n">ints</span><span class="p">.</span><span class="nf">toSet</span><span class="p">().</span><span class="nf">forEach</span><span class="p">(</span><span class="o">::</span><span class="n">remove</span><span class="p">)</span> <span class="p">}</span>
<span class="p">.</span><span class="nf">toSet</span><span class="p">()</span>
</code></pre></div></div>
<p>Visually this is not the greatest.
It’s also not really that efficient (not that we’ve been worrying about that yet).
We got here because I started with a clever-but-incorrect approach (<code class="highlighter-rouge">toList() - toSet()</code>) that I then had to refactor until it was correct.</p>
<h3 id="attempt-4">Attempt 4</h3>
<p>Fourth attempt is a chance to reset our approach.
I thought that we could partition the elements based on whether we’ve seen the value before.
A set tracks the values, and its <code class="highlighter-rouge">MutableSet.add</code> returns a boolean indicating whether the collection was mutated (i.e., has been seen before).</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">val</span> <span class="py">dupes</span><span class="p">:</span> <span class="nc">Set</span><span class="p"><</span><span class="nc">Int</span><span class="p">></span> <span class="p">=</span>
<span class="nc">HashSet</span><span class="p"><</span><span class="nc">Int</span><span class="p">>()</span>
<span class="p">.</span><span class="nf">run</span> <span class="p">{</span> <span class="n">ints</span><span class="p">.</span><span class="nf">partition</span><span class="p">(</span><span class="o">::</span><span class="n">add</span><span class="p">)</span> <span class="p">}</span>
<span class="p">.</span><span class="n">second</span>
<span class="p">.</span><span class="nf">toSet</span><span class="p">()</span>
</code></pre></div></div>
<p>This prints <code class="highlighter-rouge">[1, 3]</code> correctly.
Visually the code is just dreadful.
It’s hard to quickly discern what value is flowing from line to line.</p>
<p>Using <code class="highlighter-rouge">partition</code> was just my first intuition.
But a <code class="highlighter-rouge">partition</code> that throws away half the result has another name: a <code class="highlighter-rouge">filter</code>!</p>
<h3 id="attempt-5">Attempt 5</h3>
<p>Fifth attempt at this now using a filter.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">val</span> <span class="py">dupes</span><span class="p">:</span> <span class="nc">Set</span><span class="p"><</span><span class="nc">Int</span><span class="p">></span> <span class="p">=</span>
<span class="n">ints</span><span class="p">.</span><span class="nf">filterNot</span><span class="p">(</span><span class="nc">HashSet</span><span class="p"><</span><span class="nc">Int</span><span class="p">>()</span><span class="o">::</span><span class="n">add</span><span class="p">)</span>
<span class="p">.</span><span class="nf">toSet</span><span class="p">()</span>
</code></pre></div></div>
<p>This continues to print <code class="highlighter-rouge">[1, 3]</code> correctly.
We use <code class="highlighter-rouge">filterNot</code> because we want to keep elements where <code class="highlighter-rouge">MutableSet.add</code> returns <em>false</em>.
Visually this is pretty decent.</p>
<p>The use of <code class="highlighter-rouge">HashSet<Int>()::add</code> is what’s known as a <em>bound</em> reference.
We are specifying a function reference of <code class="highlighter-rouge">MutableSet::add</code> as our <code class="highlighter-rouge">filterNot</code> lambda, but bound to an instance of <code class="highlighter-rouge">HashSet</code> which we are creating on-the-fly.
This is an equivalent version of the above code.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">val</span> <span class="py">seen</span> <span class="p">=</span> <span class="nc">HashSet</span><span class="p"><</span><span class="nc">Int</span><span class="p">>()</span>
<span class="kd">val</span> <span class="py">dupes</span><span class="p">:</span> <span class="nc">Set</span><span class="p"><</span><span class="nc">Int</span><span class="p">></span> <span class="p">=</span>
<span class="n">ints</span><span class="p">.</span><span class="nf">filterNot</span><span class="p">(</span><span class="n">seen</span><span class="o">::</span><span class="n">add</span><span class="p">)</span>
<span class="p">.</span><span class="nf">toSet</span><span class="p">()</span>
</code></pre></div></div>
<p>The advantage of inlining the <code class="highlighter-rouge">HashSet</code> instantiation is that we don’t need to name it.<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup></p>
<h3 id="attempt-51">Attempt 5.1</h3>
<p>Finally, almost all of Kotlin’s collection extensions have <code class="highlighter-rouge">To</code>-suffixed variants which allow supplying a destination collection.
This can save you from having to add a <code class="highlighter-rouge">toSomething()</code> after an operation by instead just using that <code class="highlighter-rouge">Something</code> in the operation directly.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">val</span> <span class="py">dupes</span><span class="p">:</span> <span class="nc">Set</span><span class="p"><</span><span class="nc">Int</span><span class="p">></span> <span class="p">=</span>
<span class="n">ints</span><span class="p">.</span><span class="nf">filterNotTo</span><span class="p">(</span><span class="nc">HashSet</span><span class="p">(),</span> <span class="nc">HashSet</span><span class="p"><</span><span class="nc">Int</span><span class="p">>()</span><span class="o">::</span><span class="n">add</span><span class="p">)</span>
</code></pre></div></div>
<p>Pretty, pretty, pretty good.</p>
<h3 id="benchmarks">Benchmarks</h3>
<p>Performance is not really a concern in my usage, but let’s look anyway.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Benchmark Score Error Units
---------------------------------------------- --------- ------ -----
IntDupes.map 94.015 ± 0.469 ns/op
IntDupes.map:·gc.alloc.rate.norm 776.000 ± 0.001 B/op
IntDupes.mutableListRemove 155.744 ± 17.829 ns/op
IntDupes.mutableListRemove:·gc.alloc.rate.norm 560.000 ± 0.001 B/op
IntDupes.partition 135.693 ± 18.976 ns/op
IntDupes.partition:·gc.alloc.rate.norm 544.000 ± 0.001 B/op
IntDupes.filterNot 97.748 ± 1.055 ns/op
IntDupes.filterNot:·gc.alloc.rate.norm 504.000 ± 0.001 B/op
IntDupes.filterNotTo 39.904 ± 0.331 ns/op
IntDupes.filterNotTo:·gc.alloc.rate.norm 432.000 ± 0.001 B/op
</code></pre></div></div>
<p>So the <code class="highlighter-rouge">filterNotTo</code> winds up being the fastest and allocates the fewest bytes.
Double win!</p>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>As I’m writing this I’m realizing the <code class="highlighter-rouge">partition</code> above could have been <code class="highlighter-rouge">ints.partition(HashSet<Int>::add).second.toSet()</code>. This produces the same bytecode, but from a more compact Kotlin. <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
Intermediate collection avoidance2024-02-07T00:00:00+00:00https://jakewharton.com/intermediate-collection-avoidance<p>Given a list of users, extract their names and join them into a comma-separated list.
Kotlin’s extension functions on collections make this trivial.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">users</span><span class="p">.</span><span class="nf">map</span> <span class="p">{</span> <span class="n">it</span><span class="p">.</span><span class="n">name</span> <span class="p">}.</span><span class="nf">joinToString</span><span class="p">()</span>
</code></pre></div></div>
<p>Writing this in IntelliJ IDEA produces a “weak warning” offering advice.</p>
<blockquote>
<p>Call chain on collection type may be simplified</p>
</blockquote>
<p>An intention action will refactor the code for you to a more efficient form.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">users</span><span class="p">.</span><span class="nf">joinToString</span><span class="p">()</span> <span class="p">{</span> <span class="n">it</span><span class="p">.</span><span class="n">name</span> <span class="p">}</span>
</code></pre></div></div>
<p>Mapping the user to their name now occurs during construction of the joined string rather than as a discrete operation.
The additional iterator and intermediate collection produced by the <code class="highlighter-rouge">map</code> is eliminated.</p>
<p>This code is both shorter and faster, and the IDE helps you discover this superior form.</p>
<p>Two similar fused operations that I like but which don’t benefit from IDE advice are array and pre-sized list initialization with a lambda.</p>
<p>If we wanted to create an array of our user’s names, instead of doing</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">users</span><span class="p">.</span><span class="nf">map</span> <span class="p">{</span> <span class="n">it</span><span class="p">.</span><span class="n">name</span> <span class="p">}.</span><span class="nf">toTypedArray</span><span class="p">()</span>
</code></pre></div></div>
<p>we can use</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">Array</span><span class="p">(</span><span class="n">users</span><span class="p">.</span><span class="n">size</span><span class="p">)</span> <span class="p">{</span> <span class="n">users</span><span class="p">[</span><span class="n">it</span><span class="p">].</span><span class="n">name</span> <span class="p">}</span>
</code></pre></div></div>
<p>This again trades the intermediate iterator and collection within <code class="highlighter-rouge">map</code> for an indexed loop.
Primitive array versions are also available.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">IntArray</span><span class="p">(</span><span class="n">users</span><span class="p">.</span><span class="n">size</span><span class="p">)</span> <span class="p">{</span> <span class="n">users</span><span class="p">[</span><span class="n">it</span><span class="p">].</span><span class="n">age</span> <span class="p">}</span>
</code></pre></div></div>
<p>Arrays are not used too often.
Mostly for memory-sensitive or performance-sensitive code, or when calling out to a Java API.
Thankfully this lambda-accepting initializer is also available for pre-sized lists.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">MutableList</span><span class="p">(</span><span class="n">users</span><span class="p">.</span><span class="n">size</span><span class="p">)</span> <span class="p">{</span> <span class="n">users</span><span class="p">[</span><span class="n">it</span><span class="p">].</span><span class="n">name</span> <span class="p">}</span>
</code></pre></div></div>
<p>Use this to initialize element default values, compute elements based on the index, or derive data from another source.</p>
<p>In the case of deriving data, the source needs to support random access in order to actually result in a more efficient computation.<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup> If you use a list backed by an alternate structure (linked, persistent, etc.) performance will be abysmal. This technique works best for internal library usage and should not be used when you don’t control the original list.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Benchmark Score Error Units
--------------------------------------------- ---------- -------- -----
NamesJoinToString.map 126.582 ± 38.237 ns/op
NamesJoinToString.map:·gc.alloc.rate.norm 232.000 ± 0.001 B/op
NamesJoinToString.lambda 73.586 ± 1.960 ns/op
NamesJoinToString.lambda:·gc.alloc.rate.norm 168.000 ± 0.001 B/op
NamesToTypedArray.map 78.444 ± 22.427 ns/op
NamesToTypedArray.map:·gc.alloc.rate.norm 120.000 ± 0.001 B/op
NamesToTypedArray.lambda 10.326 ± 0.129 ns/op
NamesToTypedArray.lambda:·gc.alloc.rate.norm 40.000 ± 0.001 B/op
</code></pre></div></div>
<p>As you can see in the benchmarks above, the lambda initialization variants are both faster due to the use of indexed loops and allocate fewer bytes with no iterator or intermediate collection. We could hand-write such loops, but Kotlin’s zero-overhead functions keep our code short and sweet.</p>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>You might be aware of Compose UI’s <em>horribly</em>-named <a href="https://developer.android.com/reference/kotlin/androidx/compose/ui/util/package-summary#(kotlin.collections.List).fastMap(kotlin.Function1)">“fast” collection functions</a> which also use this strategy. <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
A stable, multiplatform Molecule 1.02023-07-19T00:00:00+00:00https://jakewharton.com/molecule-1-0This post was published externally on Cash App Code Blog. Read it at https://code.cash.app/molecule-1-0.Native UI and multiplatform Compose with Redwood2023-07-05T00:00:00+00:00https://jakewharton.com/native-ui-and-multiplatform-compose-with-redwoodThis post was published externally on Cash App Code Blog. Read it at https://code.cash.app/native-ui-and-multiplatform-compose-with-redwood.Flow testing with Turbine2023-06-21T00:00:00+00:00https://jakewharton.com/flow-testing-with-turbineThis post was published externally on Cash App Code Blog. Read it at https://code.cash.app/flow-testing-with-turbine.Using jlink to cross-compile minimal JREs2023-01-16T00:00:00+00:00https://jakewharton.com/using-jlink-to-cross-compile-minimal-jres<p><code class="highlighter-rouge">jlink</code> is a JDK tool to create bespoke, minimal JREs for your applications.
Let’s try it with a “Hello, world!” program:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Main</span> <span class="o">{</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">...</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"Hello, world!"</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>My laptop is an M1 Mac and I have downloaded the Azul Zulu JDK 19 build for it.
With the JDK I can both compile Java and then run the resulting program.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ mkdir out
$ zulu19.30.11-ca-jdk19.0.1-macosx_aarch64/bin/javac -d out in/Main.java
$ zulu19.30.11-ca-jdk19.0.1-macosx_aarch64/bin/java -cp out Main
Hello, world!
</code></pre></div></div>
<p>Azul Zulu also provides a JRE that I can use to run compiled programs.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ zulu19.30.11-ca-jre19.0.1-macosx_aarch64/bin/java -cp out Main
Hello, world!
</code></pre></div></div>
<p>Note the slight change in folder name (“jdk” → “jre”).</p>
<p>If we were shipping this to end-users it would be an easy win for binary size.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ du -hs zulu*
329M zulu19.30.11-ca-jdk19.0.1-macosx_aarch64
136M zulu19.30.11-ca-jre19.0.1-macosx_aarch64
</code></pre></div></div>
<p>But 136MiB just for “Hello, world”? Don’t tell Reddit or Hacker News!</p>
<p>Thankfully, <code class="highlighter-rouge">jlink</code> is here to help us build a minimal JRE with only what we need.
Given our program, a sibling tool, <code class="highlighter-rouge">jdeps</code>, lists the Java modules which are required.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ zulu19.30.11-ca-jdk19.0.1-macosx_aarch64/bin/jdeps \
--print-module-deps \
out/Main.class
java.base
</code></pre></div></div>
<p>Our program is so simple that it only needs the “base” module.
Now with <code class="highlighter-rouge">jlink</code> we can produce a minimal JRE.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ zulu19.30.11-ca-jdk19.0.1-macosx_aarch64/bin/jlink \
--compress 2 \
--strip-debug \
--no-header-files \
--no-man-pages \
--output zulu-hello-jre \
--add-modules java.base
$ du -hs zulu*
28M zulu-hello-jre
329M zulu19.30.11-ca-jdk19.0.1-macosx_aarch64
136M zulu19.30.11-ca-jre19.0.1-macosx_aarch64
</code></pre></div></div>
<p>28MiB won’t win any language wars, but it’s a massive 80% savings over the full JRE.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ zulu-hello-jre/bin/java -cp out Main
Hello, world!
</code></pre></div></div>
<p>We can ship it to our client and call it a day, right?</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ tar -czf hello.tgz zulu-hello-jre out
$ scp hello.tgz jw@server:
hello.tgz 100% 14MB 2.0MB/s 00:07
$ ssh jw@server "tar xzf hello.tgz && zulu-hello-jre/bin/java -cp out Main"
bash: zulu-hello-jre/bin/java: cannot execute binary file: Exec format error
</code></pre></div></div>
<p>Nope!
While the Java bytecode we compiled is platform independent, the JRE is specific to each platform and my server runs Linux x64.</p>
<p>Thankfully, <code class="highlighter-rouge">jlink</code> can operate on JDKs for different platforms.
Let’s download the Linux x64 JDK and point <code class="highlighter-rouge">jlink</code> at its Java modules using <code class="highlighter-rouge">--module-path</code>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ zulu19.30.11-ca-jdk19.0.1-macosx_aarch64/bin/jlink \
--compress 2 \
--strip-debug \
--no-header-files \
--no-man-pages \
--output zulu-hello-jre-linux-x64 \
--module-path zulu19.30.11-ca-jdk19.0.1-linux_x64/jmods
--add-modules java.base
$ du -hs zulu*
28M zulu-hello-jre
36M zulu-hello-jre-linux-x64
338M zulu19.30.11-ca-jdk19.0.1-linux_x64
329M zulu19.30.11-ca-jdk19.0.1-macosx_aarch64
136M zulu19.30.11-ca-jre19.0.1-macosx_aarch64
</code></pre></div></div>
<p>The Linux x64 JRE is a little larger than the one for my ARM Mac, but it’s still small compared to the full-size JRE.
Does it work on the client?</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ tar -czf hello-linux.tgz zulu-hello-jre-linux-x64 out
$ scp hello-linux.tgz jw@server:
hello.tgz 100% 16MB 2.1MB/s 00:08
$ ssh jw@server "tar xzf hello-linux.tgz && zulu-hello-jre-linux-x64/bin/java -cp out Main"
Hello, world!
</code></pre></div></div>
<p>It works! Now we can grab JDKs for any architecture for any platform and use our host <code class="highlighter-rouge">jlink</code> to effectively cross-compile minimal JREs for each target.</p>
<p>This is a great solution for multi-architecture Docker containers, desktop clients like JetBrains Compose UI, shipping to devices where you can’t fit a full JDK, and more.
Be sure to explore all the options on <code class="highlighter-rouge">jdeps</code> and <code class="highlighter-rouge">jlink</code> for ways to keep your runtimes small.</p>
Report card: Java 19 and the end of Kotlin2022-09-20T00:00:00+00:00https://jakewharton.com/report-card-java-19-and-the-end-of-kotlin<p>Three years ago I gave the talk <a href="/whats-new-in-java-19-the-end-of-kotlin/">“What’s new in Java 19: The end of Kotlin?”</a> which forecasted what a future Java language would look like in September 2022 when Java 19 was released.
Check your calendars, folks. It’s September 2022 right now and Java 19 was released today!</p>
<p>As expected my predictions were not perfect, but I’m pretty happy with the results.
Let’s check in with each feature and see how my predictions fared report-card style<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup>.</p>
<h3 id="local-methods">Local methods</h3>
<p>This feature allows for methods to be declared inside of other methods making them effectively private to that method.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">static</span> <span class="kt">boolean</span> <span class="nf">anyMatch</span><span class="o">(</span><span class="nc">Graph</span> <span class="n">graph</span><span class="o">,</span> <span class="nc">Predicate</span><span class="o"><</span><span class="nc">Node</span><span class="o">></span> <span class="n">predicate</span><span class="o">)</span> <span class="o">{</span>
<span class="kt">var</span> <span class="n">seen</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">HashSet</span><span class="o"><</span><span class="nc">Node</span><span class="o">>();</span>
<span class="kt">boolean</span> <span class="nf">hasMatch</span><span class="o">(</span><span class="nc">Node</span> <span class="n">node</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(!</span><span class="n">seen</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">node</span><span class="o">))</span> <span class="k">return</span> <span class="kc">false</span><span class="o">;</span> <span class="c1">// already seen</span>
<span class="k">if</span> <span class="o">(</span><span class="n">predicate</span><span class="o">.</span><span class="na">test</span><span class="o">(</span><span class="n">node</span><span class="o">))</span> <span class="k">return</span> <span class="kc">true</span><span class="o">;</span> <span class="c1">// match!</span>
<span class="k">return</span> <span class="n">node</span><span class="o">.</span><span class="na">getNodes</span><span class="o">().</span><span class="na">stream</span><span class="o">().</span><span class="na">anyMatch</span><span class="o">(</span><span class="n">n</span> <span class="o">-></span> <span class="n">hasMatch</span><span class="o">(</span><span class="n">n</span><span class="o">));</span>
<span class="o">}</span>
<span class="k">return</span> <span class="nf">hasMatch</span><span class="o">(</span><span class="n">getRoot</span><span class="o">());</span>
<span class="o">}</span>
</code></pre></div></div>
<p><strong>Grade: F</strong> 🔴</p>
<p>Working support for local methods was added to a branch in Project Amber in October 2019.
It seemed like a slam dunk, but a JEP for the feature was never created.
The branch still sits in the Project Amber repo unchanged in three years.</p>
<p>If I had to guess, all eyes in Amber are focused on pattern matching and its related features.
Hopefully someday local methods can be picked back up as a proposed feature.</p>
<h3 id="text-blocks">Text blocks</h3>
<p>A multiline string literal for when one line just isn’t enough.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"""
SELECT *
FROM users
WHERE name LIKE 'Jake %'
"""</span><span class="o">);</span>
</code></pre></div></div>
<p><strong>Grade: A</strong> 🟢</p>
<p><a href="https://openjdk.org/jeps/378">Delivered in Java 15</a>.</p>
<h3 id="records">Records</h3>
<p>A read-only type that exists solely for carrying data with strong, semantic names.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">record</span> <span class="nf">Person</span><span class="o">(</span><span class="nc">String</span> <span class="n">name</span><span class="o">,</span> <span class="kt">int</span> <span class="n">age</span><span class="o">)</span> <span class="o">{</span> <span class="o">}</span>
</code></pre></div></div>
<p><strong>Grade: A</strong> 🟢</p>
<p><a href="https://openjdk.org/jeps/395">Delivered in Java 16</a>.</p>
<h3 id="sealed-hierarchies">Sealed hierarchies</h3>
<p>Define the list of permitted subtypes of your class or interface and prevent any others.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">sealed</span> <span class="kd">interface</span> <span class="nc">Developer</span> <span class="o">{</span> <span class="o">}</span>
<span class="n">record</span> <span class="nf">Person</span><span class="o">(</span><span class="nc">String</span> <span class="n">name</span><span class="o">,</span> <span class="kt">int</span> <span class="n">age</span><span class="o">)</span> <span class="kd">extends</span> <span class="nc">Developer</span> <span class="o">{</span> <span class="o">}</span>
<span class="n">record</span> <span class="nf">Business</span><span class="o">(</span><span class="nc">String</span> <span class="n">name</span><span class="o">)</span> <span class="kd">extends</span> <span class="nc">Developer</span> <span class="o">{</span> <span class="o">}</span>
</code></pre></div></div>
<p><strong>Grade: A</strong> 🟢</p>
<p><a href="https://openjdk.org/jeps/409">Delivered in Java 17</a>.</p>
<h3 id="type-patterns">Type patterns</h3>
<p>Declare a new name to bind when a type test succeeds.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">Object</span> <span class="n">o</span> <span class="o">=</span> <span class="mi">1</span><span class="o">;</span>
<span class="k">if</span> <span class="o">(</span><span class="n">o</span> <span class="k">instanceof</span> <span class="nc">Integer</span> <span class="n">i</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">i</span> <span class="o">+</span> <span class="mi">1</span><span class="o">);</span>
<span class="o">}</span>
</code></pre></div></div>
<p><strong>Grade: A</strong> 🟢</p>
<p><a href="https://openjdk.org/jeps/394">Delivered in Java 16</a> for <code class="highlighter-rouge">instanceof</code>.
<a href="https://openjdk.org/jeps/427">Third preview in Java 19</a> for use in a <code class="highlighter-rouge">switch</code>.</p>
<h3 id="record-patterns">Record patterns</h3>
<p>Bind the component parts of a record type to local names.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">Developer</span> <span class="n">alice</span> <span class="o">=</span> <span class="nc">Person</span><span class="o">(</span><span class="s">"Alice"</span><span class="o">,</span> <span class="mi">12</span><span class="o">);</span>
<span class="k">switch</span> <span class="o">(</span><span class="n">alice</span><span class="o">)</span> <span class="o">{</span>
<span class="k">case</span> <span class="nf">Person</span><span class="o">(</span><span class="kt">var</span> <span class="n">name</span><span class="o">,</span> <span class="kt">var</span> <span class="n">age</span><span class="o">)</span> <span class="o">-></span> <span class="c1">// ...</span>
<span class="o">}</span>
</code></pre></div></div>
<p><strong>Grade: C</strong> 🟠</p>
<p><a href="https://openjdk.org/jeps/405">First preview in Java 19</a></p>
<p>Just now starting in preview and does not include things like the use of an underscore (<code class="highlighter-rouge">_</code>) as a wildcard or syntax for destructors.</p>
<h3 id="virtual-threads">Virtual threads</h3>
<p>All of your blocking calls with none of the blocking.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">try</span> <span class="o">(</span><span class="kt">var</span> <span class="n">executor</span> <span class="o">=</span> <span class="nc">Executors</span><span class="o">.</span><span class="na">newVirtualThreadPerTaskExecutor</span><span class="o">())</span> <span class="o">{</span>
<span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">count</span> <span class="o">=</span> <span class="mi">10</span><span class="o">;</span> <span class="n">count</span> <span class="o">></span> <span class="mi">0</span><span class="o">;</span> <span class="n">count</span><span class="o">--)</span> <span class="o">{</span>
<span class="n">executor</span><span class="o">.</span><span class="na">submit</span><span class="o">(()</span> <span class="o">-></span> <span class="o">{</span>
<span class="nc">Thread</span><span class="o">.</span><span class="na">sleep</span><span class="o">(</span><span class="mi">100</span> <span class="o">*</span> <span class="n">count</span><span class="o">);</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">count</span><span class="o">)</span>
<span class="o">});</span>
<span class="o">});</span>
<span class="o">}</span>
</code></pre></div></div>
<p><strong>Grade: B</strong> 🟡</p>
<p><a href="https://openjdk.org/jeps/425">First preview in Java 19</a>.</p>
<p>Nice to see it just make the cut, although expect a few previews to be needed before it’s stable.</p>
<hr />
<p>All things considered I think this is a passing report card (despite failing one elective).</p>
<p>The next three years of Java will hopefully see the completion of the items above as well as see larger efforts like Project Panama and Project Valhalla start to come to fruition.
It’s a great time to be a Java developer.</p>
<p>To the surprise of no one, Kotlin did not end.
It continued to evolve in the last three years with language features such as context receivers, sealed interfaces, and exhaustive-by-default.
It’s also a great time to be a Kotlin developer.</p>
<p>But in the end I think we can all agree on one thing: there’s no such thing as OpenJDK LTS and the best long-term version of the JDK is always the latest one. Welcome to my hill. <strong>Update to Java 19 today!</strong></p>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>A is perfect, F is fail, and there is no E. <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
Build on latest Java, test through lowest Java2022-05-17T00:00:00+00:00https://jakewharton.com/build-on-latest-java-test-through-lowest-java<p>In the past, when a new version of Java was released, I would add that version to our open source project’s CI builds.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> strategy:
matrix:
java-version:
- 8
- 9
⋮
- 17
<span class="gi">+ - 18
</span></code></pre></div></div>
<p>This ensures that each project can be built and its tests pass on every major version.</p>
<p>But this makes no sense! No user is building these projects on different versions. No user is building these projects at all. Consumers are using the pre-built <code class="highlighter-rouge">.jar</code> which we ship to Maven Central built on a single version.</p>
<p>Testing on every version, however, is something extremely valuable. Thankfully, <a href="https://docs.gradle.org/current/userguide/toolchains.html">Gradle toolchains</a> let us retain this while still only building once.</p>
<p>First, CI only has to build on a single version. We choose the latest because Java has excellent cross-compilation capabilities, and we want to be using the latest tools.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> - uses: actions/setup-java@v2
with:
distribution: 'zulu'
<span class="gd">- java-version: ${{ matrix.java-version }}
</span><span class="gi">+ java-version: 18
</span></code></pre></div></div>
<p>Second, unchanged from before, we still target whichever Java version is the lowest supported through either the <code class="highlighter-rouge">--release</code> flag or <code class="highlighter-rouge">sourceCompatibility</code>/<code class="highlighter-rouge">targetCompatibility</code> <a href="https://docs.gradle.org/7.4/userguide/building_java_projects.html#sec:java_cross_compilation">per the Gradle docs</a>.</p>
<p>And finally, we set up tests to run on every supported version.</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Normal test task runs on compile JDK.</span>
<span class="o">(</span><span class="mi">8</span><span class="o">..</span><span class="mi">17</span><span class="o">).</span><span class="na">each</span> <span class="o">{</span> <span class="n">majorVersion</span> <span class="o">-></span>
<span class="kt">def</span> <span class="n">jdkTest</span> <span class="o">=</span> <span class="n">tasks</span><span class="o">.</span><span class="na">register</span><span class="o">(</span><span class="s2">"testJdk$majorVersion"</span><span class="o">,</span> <span class="n">Test</span><span class="o">)</span> <span class="o">{</span>
<span class="n">javaLauncher</span> <span class="o">=</span> <span class="n">javaToolchains</span><span class="o">.</span><span class="na">launcherFor</span> <span class="o">{</span>
<span class="n">languageVersion</span> <span class="o">=</span> <span class="n">JavaLanguageVersion</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">majorVersion</span><span class="o">)</span>
<span class="o">}</span>
<span class="n">description</span> <span class="o">=</span> <span class="s2">"Runs the test suite on JDK $majorVersion"</span>
<span class="n">group</span> <span class="o">=</span> <span class="n">LifecycleBasePlugin</span><span class="o">.</span><span class="na">VERIFICATION_GROUP</span>
<span class="c1">// Copy inputs from normal Test task.</span>
<span class="kt">def</span> <span class="n">testTask</span> <span class="o">=</span> <span class="n">tasks</span><span class="o">.</span><span class="na">getByName</span><span class="o">(</span><span class="s2">"test"</span><span class="o">)</span>
<span class="n">classpath</span> <span class="o">=</span> <span class="n">testTask</span><span class="o">.</span><span class="na">classpath</span>
<span class="n">testClassesDirs</span> <span class="o">=</span> <span class="n">testTask</span><span class="o">.</span><span class="na">testClassesDirs</span>
<span class="o">}</span>
<span class="n">tasks</span><span class="o">.</span><span class="na">named</span><span class="o">(</span><span class="s2">"check"</span><span class="o">).</span><span class="na">configure</span> <span class="o">{</span> <span class="n">dependsOn</span><span class="o">(</span><span class="n">jdkTest</span><span class="o">)</span> <span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>This setup reduces CI burden since we only compile the main and test sources once but execute the tests on every supported version from latest to lowest.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Verification tasks
------------------
check - Runs all checks.
test - Runs the test suite.
testJdk10 - Runs the test suite on JDK 10
testJdk11 - Runs the test suite on JDK 11
testJdk12 - Runs the test suite on JDK 12
testJdk13 - Runs the test suite on JDK 13
testJdk14 - Runs the test suite on JDK 14
testJdk15 - Runs the test suite on JDK 15
testJdk16 - Runs the test suite on JDK 16
testJdk17 - Runs the test suite on JDK 17
testJdk8 - Runs the test suite on JDK 8
testJdk9 - Runs the test suite on JDK 9
</code></pre></div></div>
<p>For projects using <a href="https://openjdk.java.net/jeps/238">multi-release jars</a>, this compilation and testing setup is essential since the source sets require compiling with newer versions but testing through a lower version bound.</p>
<p>So if adding Java versions to a CI matrix is something you’ve been doing, consider switching to compile with a single Java version and instead varying your test execution instead. And if you only build and test on a single version today, adding this can ensure correctness on all versions that you support.</p>
<p>Not every project needs to test on multiple versions. If your code is mostly algorithmic you won’t gain much from doing this. But if you vary behavior based on Java version, conditionally leverage APIs on newer versions, or interact with non-public APIs then this is a best practice.</p>
<hr />
<p>P.S. Are you an Android developer? You probably keep your <code class="highlighter-rouge">compileSdk</code> high, your <code class="highlighter-rouge">minSdk</code> low(-ish), and execute instrumentation tests on a few versions between those two. Great news, you’re already following this advice as it’s always been the norm!</p>
Slope-intercept library design2022-04-05T00:00:00+00:00https://jakewharton.com/slope-intercept-library-design<p>The equation <code class="highlighter-rouge">y=mx+b</code> defines a line in slope-intercept form. The line will intercept the y-axis at the value <code class="highlighter-rouge">b</code> and for each change in <code class="highlighter-rouge">x</code> its slope (the amount the line goes up or down) will change by <code class="highlighter-rouge">m</code>.</p>
<p>Slope-intercept gives me a way to think about the design of libraries in relation to each other. The intercept is the initial cost of learning and setup for a library, and the slope is how the library’s complexity changes over time. There’s no real units here and the values are entirely subjective. Let’s try it!</p>
<h3 id="picasso">Picasso</h3>
<p>Exactly 10 years ago today I introduced Picasso internally at Square. As an image loading library for Android, its primary selling point was a low intercept. It required no real configuration and only one line of code (even in a <code class="highlighter-rouge">ListView</code> adapter).</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">Picasso</span><span class="o">.</span><span class="na">with</span><span class="o">(</span><span class="n">context</span><span class="o">).</span><span class="na">load</span><span class="o">(</span><span class="s">"https://..."</span><span class="o">).</span><span class="na">into</span><span class="o">(</span><span class="n">imageView</span><span class="o">);</span>
</code></pre></div></div>
<p>At the time this was a refreshing change from the existing libraries which required a lot of up-front and per-request configuration.</p>
<p>The downside, however, was that as your needs grow the slope of complexity also grows faster than desired. Configuring the global instance, managing multiple instances, intercepting requests, and transforming images are all possible but more difficult than if the library was designed differently.</p>
<h3 id="retrofit">Retrofit</h3>
<p>Retrofit is a declarative HTTP client abstraction for the JVM and Android. It requires configuration of a central object before you can use it to create instances of service interfaces.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">interface</span> <span class="nc">GitHubService</span> <span class="o">{</span>
<span class="nd">@GET</span><span class="o">(</span><span class="s">"users/{user}/repos"</span><span class="o">)</span>
<span class="nc">Call</span><span class="o"><</span><span class="nc">List</span><span class="o"><</span><span class="nc">Repo</span><span class="o">>></span> <span class="nf">listRepos</span><span class="o">(</span><span class="nd">@Path</span><span class="o">(</span><span class="s">"user"</span><span class="o">)</span> <span class="nc">String</span> <span class="n">user</span><span class="o">);</span>
<span class="o">}</span>
<span class="kt">var</span> <span class="n">retrofit</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">Retrofit</span><span class="o">.</span><span class="na">Builder</span><span class="o">()</span>
<span class="o">.</span><span class="na">baseUrl</span><span class="o">(</span><span class="s">"https://api.github.com/"</span><span class="o">)</span>
<span class="o">.</span><span class="na">addConverter</span><span class="o">(</span><span class="nc">MoshiJsonConverter</span><span class="o">.</span><span class="na">create</span><span class="o">())</span>
<span class="o">.</span><span class="na">build</span><span class="o">();</span>
<span class="kt">var</span> <span class="n">service</span> <span class="o">=</span> <span class="n">retrofit</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="nc">GitHubService</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
</code></pre></div></div>
<p>This up-front configuration gives Retrofit a higher intercept on the y-axis. Exposure to these APIs gives you an entrypoint to discover functionality and encourages you to manage their lifetimes in an efficient way for your usage allowing the slope of complexity to not be as steep.</p>
<h3 id="dagger">Dagger</h3>
<p>Dagger is an annotation processor-based dependency injection library for the JVM and Android. It has almost no API of its own aside from a handful of annotations. In order to use Dagger you need to learn dependency injection as a concept, learn how to build the various types to which its annotations apply, and then decide how dependency injection will fit into your architecture. It’s just about the least turn-key library I’ve ever used which gives it an extremely high conceptual intercept.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@Component</span><span class="o">(</span><span class="n">modules</span> <span class="o">=</span> <span class="o">{</span>
<span class="nc">AppModule</span><span class="o">.</span><span class="na">class</span>
<span class="o">})</span>
<span class="kd">interface</span> <span class="nc">AppComponent</span> <span class="o">{</span>
<span class="nc">App</span> <span class="nf">app</span><span class="o">();</span>
<span class="o">}</span>
<span class="nd">@Module</span>
<span class="kd">final</span> <span class="kd">class</span> <span class="nc">AppModule</span> <span class="o">{</span>
<span class="nd">@Provides</span> <span class="kd">static</span> <span class="nc">Database</span> <span class="nf">provideDatabase</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="nf">Database</span><span class="o">();</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="kd">final</span> <span class="kd">class</span> <span class="nc">App</span> <span class="o">{</span>
<span class="kd">private</span> <span class="nc">Database</span> <span class="n">database</span><span class="o">;</span>
<span class="nd">@Inject</span> <span class="nc">App</span><span class="o">(</span><span class="nc">Database</span> <span class="n">database</span><span class="o">)</span> <span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">database</span> <span class="o">=</span> <span class="n">database</span><span class="o">;</span>
<span class="o">}</span>
<span class="kt">void</span> <span class="nf">run</span><span class="o">()</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">database</span><span class="o">.</span><span class="na">getUsers</span><span class="o">());</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">...</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">AppComponent</span><span class="o">.</span><span class="na">create</span><span class="o">().</span><span class="na">app</span><span class="o">().</span><span class="na">run</span><span class="o">();</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>That’s a lot of lines to basically do <code class="highlighter-rouge">new App(new Database()).run()</code>! But of course nothing stays that simple.</p>
<p>Once you have Dagger fully integrated into a large application, adding and connecting new dependencies is as easy as adding a parameter. The library automatically figures out how to wire the two together and shares instances across pre-defined lifetimes. Its slope of complexity is extremely shallow.</p>
<p>The slope-intercept evaluations of Picasso, Retrofit, and Dagger look roughly like this:</p>
<p><img src="/static/post-image/slope-intercept-1.png" alt="" /></p>
<p>What are the units? It doesn’t matter! This is a subjective approximation of concepts.</p>
<h3 id="design">Design</h3>
<p>Slope-intercept evaluation really shines when designing new libraries. It serves as a framework for discussing the amount of complexity you front-load onto a user and the amount which is spread over the continued usage of a library.</p>
<p>Can some parameter be specified globally or should it be passed with each call? Should it be available in both locations with overriding behavior? Is there an implicit default or should you always explicitly require that it is supplied?</p>
<p>As answers to those questions are being determined, you can start to look at the library as a whole. Can multiple parameters become a composite type? Do certain parameters imply defaults for the others? Are too many concepts being pushed into global configuration rather than local?</p>
<p>And finally, you can compare your design against others to determine if you’re comfortable with its approximate slope and intercept. Picasso was built to combat image loading libraries whose complexity was that of Dagger. With the initial design I missed the mark and over-corrected to be too simple. Being closer to Retrofit would have been a much more comfortable place for the long-term health of the library.</p>
<h3 id="layering">Layering</h3>
<p>Ideally every library would have an intercept near zero and a slope near zero. That is, a library which is trivial to get started with and whose API can accommodate every use case over time without learning anything new.</p>
<p>In practice this never happens simply due to the nature of complexity. You can’t build libraries to solve non-trivial tasks while keeping the API basic and supporting myriad use cases. But what you can do is cheat by providing multiple of these hypothetical slope-intercept lines through layering.</p>
<p>Providing multiple APIs at different levels of abstraction allows solving 80% of use cases with a simple API, then 80% of the remaining 20% with a more detailed API, and then the final slice with a low-level API. Each layer is built on top of the next one with a measured reduction in API complexity.</p>
<p>In an HTTP client, for example, you can expose the declarative API for the majority, an imperative API for the minority, and then low-level protocol handlers for exotic needs. If a layer does not meet your requirements then you can always drop down to the next one for more control but also more responsibility.</p>
<p>And now what you’ve created is that exact same graph as above, except representing one library and its three layers of APIs.</p>
<p><img src="/static/post-image/slope-intercept-2.png" alt="" /></p>
<p>This is certainly no exact science. But perhaps it will help you build a better library in the future. It’s helped me!</p>
The state of managing state (with Compose)2021-11-11T00:00:00+00:00https://jakewharton.com/the-state-of-managing-state-with-composeThis post was published externally on Cash App Code Blog. Read it at https://code.cash.app/the-state-of-managing-state-with-compose.Multiplatform Compose and Gradle module metadata abuse2021-11-04T00:00:00+00:00https://jakewharton.com/multiplatform-compose-and-gradle-module-metadata-abuse<p>My primary work project for the better part of a year (named Redwood) is built on top of Compose<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup> and runs on every platform that Kotlin supports. This of course means Android, but we also have Compose running on iOS, the web, the JVM, and all other native targets. It’s truly a multiplatform Compose project<sup id="fnref:2"><a href="#fn:2" class="footnote">2</a></sup>.</p>
<p>Getting Compose to run on all these platforms isn’t as hard as you would think. The Compose runtime is written as multiplatform Kotlin code but Google only ships it compiled for Android. JetBrains goes farther by shipping versions compiled for the web and for the JVM. We simply go the whole distance and compile it for every Kotlin target, while also shipping it as a single Kotlin multiplatform artifact.</p>
<p>For a year this worked fine. However, Compose UI recently went stable which meant our Android engineers were eager to start using it in the main app (as opposed to just samples). Upon Compose UI’s introduction D8 fails with a duplicate class error:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>> Duplicate class androidx.compose.runtime.AbstractApplier found in
redwood-compose-runtime (app.cash.redwood:compose-runtime-android:0.1.0-square.15) and
runtime-1.0.0-runtime (androidx.compose.runtime:runtime:1.0.0)
</code></pre></div></div>
<p>The <code class="highlighter-rouge">androidx.compose.*</code> types are compiled into Redwood’s multiplatform Compose runtime artifact. Compose UI depends on the official Compose runtime for Android which also contains these types. Since the two artifacts have different Maven coordinates, Gradle allows both to be included in the app which eventually causes D8 to complain<sup id="fnref:3"><a href="#fn:3" class="footnote">3</a></sup>.</p>
<p>Redwood was already building Compose from the same git SHAs as Google’s release builds. Ideally we could use our own builds for every platform <em>except</em> Android, and then point at Google’s artifact solely for Android. This would allow Gradle to see the two projects as sharing a common dependency thereby de-duplicating the Compose runtime classes.</p>
<h3 id="gradle-module-metadata">Gradle module metadata</h3>
<p>The mechanism by which Kotlin multiplatform artifacts resolve the correct dependency is through Gradle’s <a href="https://docs.gradle.org/current/userguide/publishing_gradle_module_metadata.html">module metadata format</a>.</p>
<blockquote>
<p>Gradle Module Metadata is a unique format aimed at improving dependency resolution by making it multi-platform and variant-aware.</p>
</blockquote>
<p>The module metadata is a JSON document which describes the supported platforms through key/value attributes. For Redwood’s Compose runtime the module metadata looks roughly like this:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"component"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"group"</span><span class="p">:</span><span class="w"> </span><span class="s2">"app.cash.redwood"</span><span class="p">,</span><span class="w">
</span><span class="nl">"module"</span><span class="p">:</span><span class="w"> </span><span class="s2">"compose-runtime"</span><span class="p">,</span><span class="w">
</span><span class="nl">"version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0.1.0-square.15"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"variants"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"releaseApiElements-published"</span><span class="p">,</span><span class="w">
</span><span class="nl">"attributes"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"org.gradle.usage"</span><span class="p">:</span><span class="w"> </span><span class="s2">"java-api"</span><span class="p">,</span><span class="w">
</span><span class="nl">"org.jetbrains.kotlin.platform.type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"androidJvm"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"available-at"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"../../compose-runtime-android/0.1.0-square.15/compose-runtime-android-0.1.0-square.15.module"</span><span class="p">,</span><span class="w">
</span><span class="nl">"group"</span><span class="p">:</span><span class="w"> </span><span class="s2">"app.cash.redwood"</span><span class="p">,</span><span class="w">
</span><span class="nl">"module"</span><span class="p">:</span><span class="w"> </span><span class="s2">"compose-runtime-android"</span><span class="p">,</span><span class="w">
</span><span class="nl">"version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0.1.0-square.15"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"iosArm64ApiElements-published"</span><span class="p">,</span><span class="w">
</span><span class="nl">"attributes"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"artifactType"</span><span class="p">:</span><span class="w"> </span><span class="s2">"org.jetbrains.kotlin.klib"</span><span class="p">,</span><span class="w">
</span><span class="nl">"org.gradle.usage"</span><span class="p">:</span><span class="w"> </span><span class="s2">"kotlin-api"</span><span class="p">,</span><span class="w">
</span><span class="nl">"org.jetbrains.kotlin.native.target"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ios_arm64"</span><span class="p">,</span><span class="w">
</span><span class="nl">"org.jetbrains.kotlin.platform.type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"native"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"available-at"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"../../compose-runtime-iosarm64/0.1.0-square.15/compose-runtime-iosarm64-0.1.0-square.15.module"</span><span class="p">,</span><span class="w">
</span><span class="nl">"group"</span><span class="p">:</span><span class="w"> </span><span class="s2">"app.cash.redwood"</span><span class="p">,</span><span class="w">
</span><span class="nl">"module"</span><span class="p">:</span><span class="w"> </span><span class="s2">"compose-runtime-iosarm64"</span><span class="p">,</span><span class="w">
</span><span class="nl">"version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0.1.0-square.15"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="err">...</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>When a 64-bit iOS ARM target consumes the <code class="highlighter-rouge">app.cash.redwood:compose-runtime</code> dependency, Gradle will parse this JSON file and actually resolve the <code class="highlighter-rouge">app.cash.redwood:compose-runtime-iosarm64</code> artifact. It behaves somewhat like an HTTP 302 redirect by replacing the user-friendly Maven coordinate with the canonical platform-specific coordinate.</p>
<p>For an Android consumer the artifact redirect resolves to <code class="highlighter-rouge">app.cash.redwood:compose-runtime-android</code> which is one of the offending artifact coordinates seen in the duplicate class error from D8. As I mentioned above, what we want is to have this variant redirect to Google’s build of the Compose runtime and not our own.</p>
<p>We could try to alter the values in the <code class="highlighter-rouge">available-at</code> object to point to Google’s artifact, but according to the <a href="https://github.com/gradle/gradle/blob/master/subprojects/docs/src/docs/design/gradle-module-metadata-latest-specification.md#available-at-value">Gradle module metadata spec</a> the <code class="highlighter-rouge">url</code> key must also point to a metadata file which is something Google does not ship.</p>
<p>Thankfully, just below <code class="highlighter-rouge">available-at</code> in the spec, the <a href="https://github.com/gradle/gradle/blob/master/subprojects/docs/src/docs/design/gradle-module-metadata-latest-specification.md#dependencies-value"><code class="highlighter-rouge">dependencies</code> array</a> affords the ability to point at arbitrary Maven coordinates. This would allow us to define a variant with no <code class="highlighter-rouge">available-at</code> but a single <code class="highlighter-rouge">dependency</code> item to the associated Google Compose runtime artifact.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> {
"name": "releaseApiElements-published",
"attributes": {
"org.gradle.usage": "java-api",
"org.jetbrains.kotlin.platform.type": "androidJvm"
},
<span class="gd">- "available-at": {
- "url": "../../compose-runtime-android/0.1.0-square.15/compose-runtime-android-0.1.0-square.15.module",
- "group": "app.cash.redwood",
- "module": "compose-runtime-android",
- "version": "0.1.0-square.15"
- }
</span><span class="gi">+ "dependencies": [
+ {
+ "group": "androidx.compose.runtime",
+ "module": "runtime",
+ "version": {
+ "prefers": "1.0.4"
+ }
+ }
+ ]
</span> }
</code></pre></div></div>
<p>But the module metadata file is entirely generated by Gradle based on project information. How can we modify it to change the output of only a single variant?</p>
<h3 id="modifying-gradle-module-metadata">Modifying Gradle module metadata</h3>
<p>Spoiler alert: You can’t. At least not using any stable APIs that Gradle provides<sup id="fnref:4"><a href="#fn:4" class="footnote">4</a></sup>.</p>
<p>The best (only?) mechanism that I’ve found is to hook into the module metadata file generation task and perform text-based modification of the JSON immediately after it is generated.</p>
<p>First, we define a text file which contains the expected JSON contents to be replaced<sup id="fnref:5"><a href="#fn:5" class="footnote">5</a></sup>.</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"releaseApiElements-published"</span><span class="p">,</span><span class="w">
</span><span class="nl">"attributes"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"org.gradle.usage"</span><span class="p">:</span><span class="w"> </span><span class="s2">"java-api"</span><span class="p">,</span><span class="w">
</span><span class="nl">"org.jetbrains.kotlin.platform.type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"androidJvm"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"available-at"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"../../compose-runtime-android/{REDWOOD_VERSION}/compose-runtime-android-{REDWOOD_VERSION}.module"</span><span class="p">,</span><span class="w">
</span><span class="nl">"group"</span><span class="p">:</span><span class="w"> </span><span class="s2">"app.cash.redwood"</span><span class="p">,</span><span class="w">
</span><span class="nl">"module"</span><span class="p">:</span><span class="w"> </span><span class="s2">"compose-runtime-android"</span><span class="p">,</span><span class="w">
</span><span class="nl">"version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"{REDWOOD_VERSION}"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="err">,</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"releaseRuntimeElements-published"</span><span class="p">,</span><span class="w">
</span><span class="nl">"attributes"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"org.gradle.usage"</span><span class="p">:</span><span class="w"> </span><span class="s2">"java-runtime"</span><span class="p">,</span><span class="w">
</span><span class="nl">"org.jetbrains.kotlin.platform.type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"androidJvm"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"available-at"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"../../compose-runtime-android/{REDWOOD_VERSION}/compose-runtime-android-{REDWOOD_VERSION}.module"</span><span class="p">,</span><span class="w">
</span><span class="nl">"group"</span><span class="p">:</span><span class="w"> </span><span class="s2">"app.cash.redwood"</span><span class="p">,</span><span class="w">
</span><span class="nl">"module"</span><span class="p">:</span><span class="w"> </span><span class="s2">"compose-runtime-android"</span><span class="p">,</span><span class="w">
</span><span class="nl">"version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"{REDWOOD_VERSION}"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="err">,</span><span class="w">
</span></code></pre></div></div>
<p>Notice how the <code class="highlighter-rouge">{REDWOOD_VERSION}</code> placeholder is used to minimize changes to this file over time.</p>
<p>Next, define the replacement JSON in another file.</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"releaseApiElements-published"</span><span class="p">,</span><span class="w">
</span><span class="nl">"attributes"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"org.gradle.usage"</span><span class="p">:</span><span class="w"> </span><span class="s2">"java-api"</span><span class="p">,</span><span class="w">
</span><span class="nl">"org.jetbrains.kotlin.platform.type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"androidJvm"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"dependencies"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"group"</span><span class="p">:</span><span class="w"> </span><span class="s2">"androidx.compose.runtime"</span><span class="p">,</span><span class="w">
</span><span class="nl">"module"</span><span class="p">:</span><span class="w"> </span><span class="s2">"runtime"</span><span class="p">,</span><span class="w">
</span><span class="nl">"version"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"prefers"</span><span class="p">:</span><span class="w"> </span><span class="s2">"{COMPOSE_VERSION}"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="err">,</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"releaseRuntimeElements-published"</span><span class="p">,</span><span class="w">
</span><span class="nl">"attributes"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"org.gradle.usage"</span><span class="p">:</span><span class="w"> </span><span class="s2">"java-runtime"</span><span class="p">,</span><span class="w">
</span><span class="nl">"org.jetbrains.kotlin.platform.type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"androidJvm"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"dependencies"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"group"</span><span class="p">:</span><span class="w"> </span><span class="s2">"androidx.compose.runtime"</span><span class="p">,</span><span class="w">
</span><span class="nl">"module"</span><span class="p">:</span><span class="w"> </span><span class="s2">"runtime"</span><span class="p">,</span><span class="w">
</span><span class="nl">"version"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"prefers"</span><span class="p">:</span><span class="w"> </span><span class="s2">"{COMPOSE_VERSION}"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="err">,</span><span class="w">
</span></code></pre></div></div>
<p>Once again we use a special string <code class="highlighter-rouge">{COMPOSE_VERSION}</code> to minimize the need to change this file as we update to new Compose versions.</p>
<p>Finally, perform this text-based substitution immediately after the file is generated. Here the <code class="highlighter-rouge">{REDWOOD_VERSION}</code> and <code class="highlighter-rouge">{COMPOSE_VERSION}</code> placeholders are replaced with their real values.</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">tasks</span><span class="o">.</span><span class="na">named</span><span class="o">(</span><span class="s2">"generateMetadataFileForKotlinMultiplatformPublication"</span><span class="o">).</span><span class="na">configure</span> <span class="o">{</span>
<span class="n">doLast</span> <span class="o">{</span>
<span class="n">String</span> <span class="n">find</span> <span class="o">=</span> <span class="n">file</span><span class="o">(</span><span class="s1">'module_find.txt'</span><span class="o">).</span><span class="na">text</span><span class="o">.</span><span class="na">replace</span><span class="o">(</span><span class="s1">'{REDWOOD_VERSION}'</span><span class="o">,</span> <span class="n">version</span><span class="o">)</span>
<span class="n">String</span> <span class="n">replace</span> <span class="o">=</span> <span class="n">file</span><span class="o">(</span><span class="s1">'module_replace.txt'</span><span class="o">).</span><span class="na">text</span><span class="o">.</span><span class="na">replace</span><span class="o">(</span><span class="s1">'{COMPOSE_VERSION}'</span><span class="o">,</span> <span class="n">versions</span><span class="o">.</span><span class="na">compose</span><span class="o">)</span>
<span class="n">File</span> <span class="n">file</span> <span class="o">=</span> <span class="n">outputFile</span><span class="o">.</span><span class="na">get</span><span class="o">().</span><span class="na">getAsFile</span><span class="o">()</span>
<span class="n">String</span> <span class="n">text</span> <span class="o">=</span> <span class="n">file</span><span class="o">.</span><span class="na">text</span>
<span class="kt">int</span> <span class="n">start</span> <span class="o">=</span> <span class="n">text</span><span class="o">.</span><span class="na">indexOf</span><span class="o">(</span><span class="n">find</span><span class="o">)</span>
<span class="k">if</span> <span class="o">(</span><span class="n">start</span> <span class="o">==</span> <span class="o">-</span><span class="mi">1</span><span class="o">)</span> <span class="o">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">RuntimeException</span><span class="o">(</span><span class="s2">"Unable to locate module_find.txt contents in module JSON ($file)"</span><span class="o">)</span>
<span class="o">}</span>
<span class="kt">int</span> <span class="n">end</span> <span class="o">=</span> <span class="n">start</span> <span class="o">+</span> <span class="n">find</span><span class="o">.</span><span class="na">length</span><span class="o">()</span>
<span class="n">String</span> <span class="n">newText</span> <span class="o">=</span> <span class="n">text</span><span class="o">.</span><span class="na">substring</span><span class="o">(</span><span class="mi">0</span><span class="o">,</span> <span class="n">start</span><span class="o">)</span> <span class="o">+</span> <span class="n">replace</span> <span class="o">+</span> <span class="n">text</span><span class="o">.</span><span class="na">substring</span><span class="o">(</span><span class="n">end</span><span class="o">)</span>
<span class="n">file</span><span class="o">.</span><span class="na">text</span> <span class="o">=</span> <span class="n">newText</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>This is some very hacky code, but any unexpected changes to the module metadata format will cause a build failure allowing you to reevaluate the approach. Perhaps in the future Gradle will support this type of transformation <a href="https://github.com/gradle/gradle/issues/18862">with a stable public API</a>.</p>
<p>This simple text substitution solves the original duplicate class problem today. And it does so in a way which does not require the consumer to understand the nuances of how the Compose runtime is built.</p>
<hr />
<p>Despite solving the issue for Android builds, we still have the duplicate class problem for the other platforms on which multiple Compose-based projects can be used. If you happened to use Redwood on the JVM with JetBrains’ Compose for Desktop you would have two copies of the Compose runtime (potentially built from different versions). The same is true for targeting the web and using JetBrains’ Compose for Web.</p>
<p>Google really should be shipping the Compose runtime as a proper multiplatform artifact for all Kotlin targets to remedy this situation. Unfortunately their Kotlin multiplatform story is a few years behind the community’s need and the prospect of this happening anytime soon is very unlikely. The best we can hope for now is JetBrains to ship a proper multiplatform artifact of the Compose runtime with the same versioning as Google’s and using this hack to point the Android variant at Google’s binary. Then everyone in the multiplatform Compose space could standardize on their artifacts.</p>
<p>Until then, however, we’ll continue the imperfect practice of building our own Compose runtime for Redwood and pointing to Google’s artifact for Android<sup id="fnref:6"><a href="#fn:6" class="footnote">6</a></sup>.</p>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>Obligatory: <a href="/a-jetpack-compose-by-any-other-name/">I mean Compose and <strong>NOT</strong> Compose UI</a>! <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
<li id="fn:2">
<p>Continuing with the poor naming surrounding Compose, JetBrains has a project called “Compose Multiplatform” which is not fully multiplatform nor fully ports Compose UI to each supported platform. Our project is “just” the Compose runtime (not Compose UI) but running fully multiplatform. <a href="#fnref:2" class="reversefootnote">↩</a></p>
</li>
<li id="fn:3">
<p>Unlike the JVM whose classpath is a set of jars which each contain classes where the first wins, Android’s classpath is a single set of classes in which duplicates are not supported (because of the dex file format). <a href="#fnref:3" class="reversefootnote">↩</a></p>
</li>
<li id="fn:4">
<p>As of Gradle 7.2. <a href="#fnref:4" class="reversefootnote">↩</a></p>
</li>
<li id="fn:5">
<p>Omitted from the earlier example, some variants have both an “api” and “runtime” entry. <a href="#fnref:5" class="reversefootnote">↩</a></p>
</li>
<li id="fn:6">
<p>We also have to build the Compose Kotlin compiler plugin for native because of how the Kotlin/Native compiler works. <a href="https://issuetracker.google.com/issues/205021616">Google could ship it</a>, or <a href="https://youtrack.jetbrains.com/issue/KT-27683">JetBrains could make the existing plugins work for native</a>. <a href="#fnref:6" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
Gradle dependency license validation2021-06-08T00:00:00+00:00https://jakewharton.com/gradle-dependency-license-validationThis post was published externally on Cash App Code Blog. Read it at https://code.cash.app/gradle-dependency-license-validation.Case-insensitive filesystems considered harmful (to me)2021-06-04T00:00:00+00:00https://jakewharton.com/case-insensitive-filesystems-considered-harmful-to-me<p>Having been burned by case-insensitive filesystem bugs one too many times, I long ago switched my development folder to a case-sensitive filesystem partition on my otherwise case-insensitive Mac. Unfortunately this can actually work against me as I interact with the computers of coworkers and service providers which use the default. Well I was burned again, and this is the tale!</p>
<p>I’ve been working on two projects based on <a href="https://developer.android.com/jetpack/compose">Jetpack Compose</a><sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup> which require me to recompile its sources. Despite building them unmodified, I still run its tests against my compiled version to ensure this core functionality of my project behaves as expected. However, both of my projects recently started experiencing test failures on CI, and it was the same, single test failing on both projects.</p>
<p>The first project <a href="https://github.com/JakeWharton/mosaic/runs/2547311635">failed about a month ago</a> when I added a MacOS worker in addition to the Linux worker to build a JNI library. Being so focused on the JNI compilation, I figured the Compose failure was a flake or something wrong with my setup. Its failure was:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>androidx.compose.runtime.CompositionTests[jvm] > testInsertOnMultipleLevels[jvm] FAILED
java.lang.NoClassDefFoundError: androidx/compose/runtime/CompositionTests$testInsertOnMultipleLevels$1$item$1 (wrong name: androidx/compose/runtime/CompositionTests$testInsertOnMultipleLevels$1$Item$1)
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
⋮
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
at androidx.compose.runtime.CompositionTests$testInsertOnMultipleLevels$1.invokeSuspend$Item(CompositionTests.kt:2055)
</code></pre></div></div>
<p>Like I said I didn’t look too closely at this output and assumed it was my own fault.</p>
<p>The second project (which is not open source yet) started failing yesterday when I added a Windows worker to publish new targets for its Kotlin multiplatform library. Notably, the project already had a MacOS worker, and the PR to add the Windows worker did see both workers succeed. The merge commit, however, failed with an exception on the Windows worker which looked awfully familiar:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>androidx.compose.runtime.CompositionTests[jvm] > testInsertOnMultipleLevels[jvm] FAILED
java.lang.NoClassDefFoundError: androidx/compose/runtime/CompositionTests$testInsertOnMultipleLevels$1$Item$1 (wrong name: androidx/compose/runtime/CompositionTests$testInsertOnMultipleLevels$1$item$1)
at java.lang.ClassLoader.defineClass1(Native Method)
⋮
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at androidx.compose.runtime.CompositionTests$testInsertOnMultipleLevels$1.invokeSuspend$Item(CompositionTests.kt:2055)
</code></pre></div></div>
<p>“It’s the same exception!”, my brain thought. But if you look closely it <em>is</em> the same but it’s also different. In this case we tried to load <code class="highlighter-rouge">CompositionTests$testInsertOnMultipleLevels$1$Item$1</code> (note the uppercase “i” in <code class="highlighter-rouge">Item</code>) but found a class named <code class="highlighter-rouge">CompositionTests$testInsertOnMultipleLevels$1$item$1</code> (note the lowercase “i” in <code class="highlighter-rouge">item</code>). This is in contrast to the first exception above where the “item” casing is reversed.</p>
<p>Cracking open <code class="highlighter-rouge">CompositionTests</code> we can look at the <code class="highlighter-rouge">testInsertOnMultipleLevels</code> method and see the source of this class:</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fun</span> <span class="nf">testInsertOnMultipleLevels</span><span class="p">()</span> <span class="p">=</span> <span class="nf">compositionTest</span> <span class="p">{</span>
<span class="c1">// …code…</span>
<span class="k">fun</span> <span class="nf">Item</span><span class="p">(</span><span class="n">number</span><span class="p">:</span> <span class="nc">Int</span><span class="p">,</span> <span class="n">numbers</span><span class="p">:</span> <span class="nc">List</span><span class="p"><</span><span class="nc">Int</span><span class="p">>)</span> <span class="p">{</span>
<span class="nc">Linear</span> <span class="p">{</span>
<span class="c1">// --> This lambda is the source! <--</span>
<span class="c1">// …code…</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c1">// …code…</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The anonymous lambda passed to <code class="highlighter-rouge">compositionTest</code> becomes <code class="highlighter-rouge">$1</code>, the nested <code class="highlighter-rouge">Item</code> function becomes <code class="highlighter-rouge">$Item</code>, and the lambda passed to <code class="highlighter-rouge">Linear</code> becomes another <code class="highlighter-rouge">$1</code> producing the final class name of <code class="highlighter-rouge">CompositionTests$testInsertOnMultipleLevels$1$Item$1</code>.</p>
<p>This all seems fine, though. So how could the name of the class for the function change casing from <code class="highlighter-rouge">Item</code> to <code class="highlighter-rouge">item</code>?</p>
<p>Thankfully, with the investigative powers of <a href="https://medium.com/@isaac.udy_90859">Isaac Udy</a> helping, we stumbled upon more code further down the function:</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fun</span> <span class="nf">testInsertOnMultipleLevels</span><span class="p">()</span> <span class="p">=</span> <span class="nf">compositionTest</span> <span class="p">{</span>
<span class="c1">// …code…</span>
<span class="k">fun</span> <span class="nf">Item</span><span class="p">(</span><span class="n">number</span><span class="p">:</span> <span class="nc">Int</span><span class="p">,</span> <span class="n">numbers</span><span class="p">:</span> <span class="nc">List</span><span class="p"><</span><span class="nc">Int</span><span class="p">>)</span> <span class="p">{</span>
<span class="nc">Linear</span> <span class="p">{</span>
<span class="c1">// …code…</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c1">// …code…</span>
<span class="k">fun</span> <span class="nc">MockViewValidator</span><span class="p">.</span><span class="nf">item</span><span class="p">(</span><span class="n">number</span><span class="p">:</span> <span class="nc">Int</span><span class="p">,</span> <span class="n">numbers</span><span class="p">:</span> <span class="nc">List</span><span class="p"><</span><span class="nc">Int</span><span class="p">>)</span> <span class="p">{</span>
<span class="nc">Linear</span> <span class="p">{</span>
<span class="c1">// …code…</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c1">// …code…</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The class generation in this second nested function follow a similar formula to the first. The anonymous lambda passed to <code class="highlighter-rouge">compositionTest</code> once again becomes <code class="highlighter-rouge">$1</code>, the nested <code class="highlighter-rouge">MockViewValidator.item</code> function becomes <code class="highlighter-rouge">$item</code>, and the lambda passed to <code class="highlighter-rouge">Linear</code> becomes another <code class="highlighter-rouge">$1</code> producing the final class name of <code class="highlighter-rouge">CompositionTests$testInsertOnMultipleLevels$1$item$1</code>.</p>
<p>And there it is. The lambda inside first function produces a class named <code class="highlighter-rouge">CompositionTests$testInsertOnMultipleLevels$1$Item$1</code> which is written to <code class="highlighter-rouge">CompositionTests$testInsertOnMultipleLevels$1$Item$1.class</code> on the filesystem. The lambda inside the second function produces a class named <code class="highlighter-rouge">CompositionTests$testInsertOnMultipleLevels$1$item$1</code> which is written to <code class="highlighter-rouge">CompositionTests$testInsertOnMultipleLevels$1$item$1.class</code> on the filesystem. Except on a case-insensitive filesystem, <em>those are the same file!</em></p>
<p>To be clear, the problematic steps are this:</p>
<ol>
<li>The build system cleans the output directory giving us a blank slate on the filesystem.</li>
<li>The Kotlin compiler generates the class <code class="highlighter-rouge">CompositionTests$testInsertOnMultipleLevels$1$Item$1</code>.</li>
<li>The Kotlin compiler opens the <code class="highlighter-rouge">CompositionTests$testInsertOnMultipleLevels$1$Item$1.class</code> file (which does not exist and is created), writes the bytecode for <code class="highlighter-rouge">CompositionTests$testInsertOnMultipleLevels$1$Item$1</code>, and closes the file.</li>
<li>The Kotlin compiler generates the class <code class="highlighter-rouge">CompositionTests$testInsertOnMultipleLevels$1$item$1</code>.</li>
<li>The Kotlin compiler opens the <code class="highlighter-rouge">CompositionTests$testInsertOnMultipleLevels$1$item$1.class</code> file (but the filesystem sees <code class="highlighter-rouge">CompositionTests$testInsertOnMultipleLevels$1$Item$1.class</code> as an existing match and opens it as an existing file), writes the bytecode for <code class="highlighter-rouge">CompositionTests$testInsertOnMultipleLevels$1$item$1</code>, and closes the file.</li>
</ol>
<p>When the project builds on my machine the non-standard, case-sensitive filesystem sees those as separate files and the failure does not occur. On MacOS- and Windows-based CI workers with their filesystem defaults, however, they’re seen as the same and one overwrites the other. This is what leads to the class name of the second appearing in the file name of the first.</p>
<p>The fix here is easy: rename one of the functions to produce different names. And in an ironic twist of timing, JetBrains <a href="https://android.googlesource.com/platform/frameworks/support/+/f705520d29e250a762c7c8ba354715e3def6fcde%5E!/">made the exact fix</a> to Compose just 12 hours ago.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gd">-fun MockViewValidator.item(number: Int, numbers: List<Int>) {
</span><span class="gi">+fun MockViewValidator.validateItem(number: Int, numbers: List<Int>) {
</span> Linear {
// …code…
}
}
</code></pre></div></div>
<p>A simple git submodule update and all my problems are now solved.</p>
<p>Or are they?</p>
<p>This is not the first time I have had this problem, and it likely won’t be the last. I would like to make the argument that this is a Kotlin compiler bug. Regardless of whether you are targeting a case-insensitive filesystem, the Kotlin compiler could avoid this entire class of problem by further mangling the name of this otherwise unnamed type to avoid case-insensitive collision.</p>
<p>You can trivially reproduce this if you have a case-insensitive filesystem:</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Hey</span>
<span class="kd">class</span> <span class="err">hey
</span></code></pre></div></div>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ kotlinc Hey.kt
$ ls Hey*
Hey.class Hey.kt
</code></pre></div></div>
<p>And a minimal reproducer for the more cryptic cause in this post would be:</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fun</span> <span class="nf">complex</span><span class="p">()</span> <span class="p">=</span> <span class="nf">run</span> <span class="p">{</span>
<span class="k">fun</span> <span class="nf">Nested</span><span class="p">()</span> <span class="p">{</span>
<span class="nf">run</span> <span class="p">{</span> <span class="nf">println</span><span class="p">(</span><span class="s">"Nested"</span><span class="p">)</span> <span class="p">}</span>
<span class="p">}</span>
<span class="k">fun</span> <span class="nc">String</span><span class="p">.</span><span class="nf">nested</span><span class="p">()</span> <span class="p">{</span>
<span class="nf">run</span> <span class="p">{</span> <span class="nf">println</span><span class="p">(</span><span class="s">"String.nested"</span><span class="p">)</span> <span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">fun</span> <span class="nf">run</span><span class="p">(</span><span class="n">lambda</span><span class="p">:</span> <span class="p">()</span> <span class="p">-></span> <span class="nc">Unit</span><span class="p">)</span> <span class="p">=</span> <span class="nf">lambda</span><span class="p">()</span>
</code></pre></div></div>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ kotlinc Complex.kt
$ ls Complex*
Complex.kt ComplexKt$complex$1$Nested$1.class ComplexKt$complex$1.class ComplexKt.class
</code></pre></div></div>
<p>I have filed <a href="https://youtrack.jetbrains.com/issue/KT-47123">KT-47123</a> to advocate that the compiler should automatically prevent this from happening.</p>
<p>Hey Java users you’re not totally immune either!</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Hey</span> <span class="o">{}</span>
<span class="kd">class</span> <span class="nc">hey</span> <span class="o">{}</span>
</code></pre></div></div>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ javac Hey.java
$ ls Hey*
Hey.class Hey.java
</code></pre></div></div>
<p>I’m confident that <em>this</em> year will finally be the year of the Linux desktop to solve all these problems with its case-sensitive-by-default filesystems, right? But until then, having tools which are smarter about filesystem interaction in a world where both case-sensitive and case-insensitive variants exist would go a long way to reducing developer headaches like this.</p>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>Obligatory: <a href="/a-jetpack-compose-by-any-other-name/">I mean Compose and <strong>NOT</strong> Compose UI</a>! <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
Cross-compiling static Rust binaries in Docker for Raspberry Pi2021-05-27T00:00:00+00:00https://jakewharton.com/cross-compiling-static-rust-binaries-in-docker-for-raspberry-pi<p>Earlier this year I built a web-based garage door controller using Rust for the Raspberry Pi called <a href="https://github.com/JakeWharton/NormallyClosed">Normally Closed</a>. My deployment includes a Pi 3b and a Pi Zero which are ARMv7 and ARMv6 devices, respectively. I deploy services with Docker and wanted to continue using it here for simplicity.</p>
<p>Getting all of this set up and working together was not easy. There’s also a lot of words in that title which might not mean much to you. That’s okay! This is the blog post that I needed two weeks ago, so let’s take a look at the steps required to accomplish this task.</p>
<h3 id="cross-compiling-and-static-linking">Cross-compiling and static linking</h3>
<p>Rust has excellent facilities for cross-compiling and static linking through Cargo. I got started following <a href="https://medium.com/swlh/compiling-rust-for-raspberry-pi-arm-922b55dbb050#2b8a">this guide</a> on cross-compiling Rust for the Raspberry Pi.</p>
<p>The guide recommends using the <code class="highlighter-rouge">armv7-unknown-linux-gnueabihf</code> Rust target which would support my Pi 3b. For the Pi Zero we can infer from <a href="https://doc.rust-lang.org/nightly/rustc/platform-support.html">Rust’s platform support list</a> that we need <code class="highlighter-rouge">arm-unknown-linux-gnueabihf</code>. However, these targets dynamically link against GNU libc whereas I wanted to statically link with musl. Referencing the platform support list again we can find the <code class="highlighter-rouge">armv7-unknown-linux-musleabihf</code> and <code class="highlighter-rouge">arm-unknown-linux-musleabihf</code> targets to use instead.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ rustup target add armv7-unknown-linux-musleabihf
$ rustup target add arm-unknown-linux-musleabihf
</code></pre></div></div>
<p>In addition to the target, the linker needs to be changed since it otherwise will use the one from your machine and its architecture. This can be specified in <code class="highlighter-rouge">.cargo/config</code>:</p>
<div class="language-toml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">[target.armv7-unknown-linux-musleabihf]</span>
<span class="py">linker</span> <span class="p">=</span> <span class="s">"arm-linux-gnueabihf-ld"</span>
<span class="nn">[target.arm-unknown-linux-musleabihf]</span>
<span class="py">linker</span> <span class="p">=</span> <span class="s">"arm-linux-gnueabihf-ld"</span>
</code></pre></div></div>
<p>On Ubuntu you can install this linker by running <code class="highlighter-rouge">sudo apt install gcc-arm-linux-gnueabihf</code>. On my Mac I was able to install it with <code class="highlighter-rouge">brew install arm-linux-gnueabihf-binutils</code>. (Despite using the musl compilation target, the GNU linker will still work.)</p>
<p>This is enough to compile working binaries!</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cargo build --target armv7-unknown-linux-musleabihf --release
$ cargo build --target arm-unknown-linux-musleabihf --release
</code></pre></div></div>
<h4 id="bonus-smaller-binaries">Bonus: Smaller binaries</h4>
<p>I really like to make my binaries and Docker containers as small as possible. Once again Rust’s Cargo gives us a simple mechanism to achieve this in our <code class="highlighter-rouge">Cargo.toml</code>:</p>
<div class="language-toml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">[profile.release]</span>
<span class="py">opt-level</span> <span class="p">=</span> <span class="s">'z'</span>
<span class="py">lto</span> <span class="p">=</span> <span class="kc">true</span>
<span class="py">codegen-units</span> <span class="p">=</span> <span class="mi">1</span>
</code></pre></div></div>
<p>Setting <code class="highlighter-rouge">opt-level</code> to <code class="highlighter-rouge">z</code> instructs the compiler to favor a smaller binary size over performance. This will not be appropriate for anything CPU-intensive, but for a web server which will only see a few interactions <em>per year</em> we don’t require maximum performance.</p>
<p>LTO is short for “link-time optimization” which performs optimization on the whole program rather than locally on individual functions. By virtue of analyzing the whole program it also improves the ability to remove dead code.</p>
<p>Finally, the <code class="highlighter-rouge">codegen-units</code> setting reduces the parallelism of Cargo to allow compilation to occur in a single unit and be optimized as a single unit. This allows compilation and optimization to have the maximum impact by always seeing the entire program.</p>
<h3 id="building-inside-docker">Building inside Docker</h3>
<p>Having starting with only the Pi 3b and needing ARMv7, getting the build going in Docker was not too difficult. We basically just run the commands from above to produce the binary and then copy that into an Alpine container.</p>
<div class="language-dockerfile highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">FROM</span><span class="s"> rust:1.52.1 AS rust</span>
<span class="k">RUN </span>rustup target add armv7-unknown-linux-musleabihf
<span class="k">RUN </span>apt-get update <span class="o">&&</span> apt-get <span class="nt">-y</span> <span class="nb">install </span>binutils-arm-linux-gnueabihf
<span class="k">WORKDIR</span><span class="s"> /app</span>
<span class="k">COPY</span><span class="s"> .cargo ./.cargo</span>
<span class="k">COPY</span><span class="s"> Cargo.toml Cargo.lock .rustfmt.toml ./</span>
<span class="k">COPY</span><span class="s"> src ./src</span>
<span class="k">RUN </span>cargo build <span class="nt">--release</span> <span class="nt">--target</span> armv7-unknown-linux-musleabihf
<span class="k">FROM</span><span class="s"> --platform linux/arm alpine:3.12</span>
<span class="k">WORKDIR</span><span class="s"> /app</span>
<span class="k">COPY</span><span class="s"> --from=rust /app/target/armv7-unknown-linux-musleabihf/release/normally-closed ./</span>
<span class="c"># ENTRYPOINT setup...</span>
</code></pre></div></div>
<p>This container can be built with the regular <code class="highlighter-rouge">docker build .</code> command.</p>
<p>Adding a second architecture complicates things significantly. Docker does support containers which are built for multiple architectures through <a href="https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images">Docker buildx</a>.</p>
<p>However, unlike the buildx examples, we cannot naively run <code class="highlighter-rouge">docker buildx build --platform linux/arm/v7,linux/arm/v6 .</code> and have it just work. For one, the Rust container is not available for those architectures. But even if it were, we still need to specify the custom compilation target per architecture due to our desire to use musl.</p>
<p>The first step towards making this work is having the Rust container always use the architecture of the machine on which it is running. This is similar to having run Cargo directly on our machine before, and it works because Rust is already allowing us to cross-compile to ARM.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gd">-FROM rust:1.52.1 AS rust
</span><span class="gi">+FROM --platform=$BUILDPLATFORM rust:1.52.1 AS rust
</span> RUN rustup target add armv7-unknown-linux-musleabihf
⋮
</code></pre></div></div>
<p>The <code class="highlighter-rouge">BUILDPLATFORM</code> argument <a href="https://docs.docker.com/engine/reference/builder/#automatic-platform-args-in-the-global-scope">is documented</a> as one which is available by default in the global scope of a Dockerfile.</p>
<p>With Rust always running on our host architecture we still need to vary the Rust target which is used for cross-compilation. Also present in the list of default arguments in Docker is <code class="highlighter-rouge">TARGETPLATFORM</code> which will contain either <code class="highlighter-rouge">linux/arm/v7</code> or <code class="highlighter-rouge">linux/arm/v6</code> in our case. We can use a <code class="highlighter-rouge">case</code> statement to determine the associated Rust target.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> FROM --platform=$BUILDPLATFORM rust:1.52.1 AS rust
<span class="gi">+ARG TARGETPLATFORM
+RUN case "$TARGETPLATFORM" in \
+ "linux/arm/v7") echo armv7-unknown-linux-musleabihf > /rust_target.txt ;; \
+ "linux/arm/v6") echo arm-unknown-linux-musleabihf > /rust_target.txt ;; \
+ *) exit 1 ;; \
+esac
</span> RUN rustup target add armv7-unknown-linux-musleabihf
⋮
</code></pre></div></div>
<p>We write the value to a file since <code class="highlighter-rouge">export</code> does not work and there’s not really another mechanism for passing data between build steps. A read of that file replaces the hard-coded targets in the steps that follow.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> ⋮
esac
<span class="gd">-RUN rustup target add armv7-unknown-linux-musleabihf
</span><span class="gi">+RUN rustup target add $(cat /rust_target.txt)
</span> RUN apt-get update && apt-get -y install binutils-arm-linux-gnueabihf
⋮
COPY src ./src
<span class="gd">-RUN cargo build --release --target armv7-unknown-linux-musleabihf
</span><span class="gi">+RUN cargo build --release --target $(cat /rust_target.txt)
</span></code></pre></div></div>
<p>The Alpine build stage also references the target in the source folder of the binary. Rather than worry about passing along the file which holds this value, an easy workaround is to copy the binary to a location which does not contain the target name.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> ⋮
RUN cargo build --release --target $(cat /rust_target.txt)
<span class="gi">+# Move the binary to a location free of the target since that is not available in the next stage.
+RUN cp target/$(cat /rust_target.txt)/release/normally-closed .
</span> ⋮
</code></pre></div></div>
<p>The Alpine build stage can now remove the target platform and copy from the new location.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> ⋮
<span class="gd">-FROM --platform linux/arm alpine:3.12
</span><span class="gi">+FROM alpine:3.12
</span> WORKDIR /app
<span class="gd">-COPY --from=rust /app/target/armv7-unknown-linux-musleabihf/release/normally-closed ./
</span><span class="gi">+COPY --from=rust /app/normally-closed ./
</span> # ENTRYPOINT setup...
</code></pre></div></div>
<p>At this point we have a fully-working, cross-compiling, static-linking, multi-architecture Docker container built from Rust!</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ docker buildx build --platform linux/arm/v7,linux/arm/v6 .
[+] Building 147.9s (34/34) FINISHED
</code></pre></div></div>
<p>You can see the result reflected on <a href="https://hub.docker.com/r/jakewharton/normally-closed/tags">the Docker Hub listing</a> as compared to the latest release:</p>
<p><a href="/static/post-image/normally-closed-docker-hub.png"><img src="/static/post-image/normally-closed-docker-hub.png" alt="Screenshot of Docker Hub showing the container has two architectures" /></a></p>
<p>The full and final <code class="highlighter-rouge">Dockerfile</code> can be found <a href="https://github.com/JakeWharton/NormallyClosed/blob/a21e4de89ef90417a99cadf75c2b6297eda35735/Dockerfile">here</a> for reference. The repository also contains GitHub Actions setup for building the standalone binaries as well as the multi-architecture Docker container.</p>
<p>Hopefully this helps someone! It was a couple nights of piecing together all the steps for me. And hey if you have a garage door and a spare Pi lying around maybe try out <a href="https://github.com/JakeWharton/NormallyClosed">Normally Closed</a>!</p>
Migrating from Burst to TestParameterInjector2021-04-15T00:00:00+00:00https://jakewharton.com/migrating-from-burst-to-testparameterinjectorThis post was published externally on Cash App Code Blog. Read it at https://code.cash.app/migrating-from-burst-to-testparameterinjector.Integration verbosity and good layering2021-04-07T00:00:00+00:00https://jakewharton.com/integration-verbosity-and-good-layering<p>One of my favorite non-features from building <a href="https://developer.android.com/topic/libraries/view-binding">view binding</a> is that it lacks integration with
activities or fragments. If you use view binding with activities or fragments, however, this fact
might be to your disdain. Every activity using view binding is forced to do something along the
lines of:</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">override</span> <span class="k">fun</span> <span class="nf">onCreate</span><span class="p">(</span><span class="n">savedInstanceState</span><span class="p">:</span> <span class="nc">Bundle</span><span class="p">)</span> <span class="p">{</span>
<span class="k">super</span><span class="p">.</span><span class="nf">onCreate</span><span class="p">(</span><span class="n">savedInstanceState</span><span class="p">)</span>
<span class="kd">val</span> <span class="py">binding</span> <span class="p">=</span> <span class="nc">ProfileViewBinding</span><span class="p">.</span><span class="nf">inflate</span><span class="p">(</span><span class="n">layoutInflater</span><span class="p">)</span>
<span class="nf">setContentView</span><span class="p">(</span><span class="n">binding</span><span class="p">.</span><span class="n">root</span><span class="p">)</span>
<span class="c1">// Do stuff with 'binding'</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This is textbook verbosity, and some would argue boilerplate. It only gets worse with fragments
(due to their poor design and to no specific fault of view binding which works the same as any
<code class="highlighter-rouge">View</code> reference).</p>
<p>View binding exists at a different layer of abstraction than is appropriate for integration with
higher-level components like activities or fragments. It serves as a type-safe representation of a
schema declared in an XML file and that’s it. It has no more knowledge of activities and fragments
than the associated <code class="highlighter-rouge">R.layout.profile_view</code> integer does.</p>
<p>Higher-level libraries like androidx.activity and androidx.fragment have integrations with those
<code class="highlighter-rouge">R.layout</code> integers. If you’re upset that view binding has no turn-key solution for activities and
fragments then this is the tree you should be barking up.</p>
<p>View binding wasn’t built with verbosity in mind.
Hell, it’s not even <em>that</em> verbose.
It ended up this way because it’s the design the layer of abstraction it operates at demands.</p>
<p>The same pattern occurs in some of my other favorite libraries.
Dagger offers you nothing and requires that you build up the dependency injectors, their hierarchy, and their lifecycle entirely yourself.
SQLDelight makes you specify database info in the build configuration and a database driver in the runtime API.
RecylerView requires at minimum an adapter subtype and to choose and configure a layout manager.
The layer at which these tools operate is sufficiently general such that their good design requires them to hoist a bunch of decisions to their caller.</p>
<p>If Dagger was more opinionated about integration with Android it’s hard to imagine Hilt could have been built as it is today.
If SQLDelight was more opinionated about talking to SQLite on Android it’s hard to imagine it could support talking to SQLite, MySQL, or Postgres on any platform as it does today.
If RecyclerView was more opinionated about layout managers or adapters it’s hard to imagine ViewPager 2 could have been built as it is today.</p>
<p>I certainly bear many scars of layering mistakes in my library past.</p>
<p>Picasso shipped with a global, static <code class="highlighter-rouge">get()</code> method so that image loading could be a one-liner with no setup.
But what if you need to configure the HTTP client or set cache policy or need two different versions of those things?
Libraries even shipped on top of Picasso using <code class="highlighter-rouge">get()</code> and assuming it would behave a certain way.
It is a mistake to assume there will be only one configuration even if it is true 99% of the time.
<del>Half-life 3</del> Picasso 3 (if it ever ships) corrects this mistake by only offering instance-based APIs.
If you want a global instance it’s only one line of code, and it’s now your decision to make.</p>
<p>Retrofit 1 shipped with a Gson dependency that was enabled by default.
You could still swap in a different converter if you wanted, but Gson would always be there.
It is a mistake to assume someone will be speaking JSON and that they will want to use Gson even if that was true (then) 99% of the time.
We know literally nothing about the enclosing application or the server it’s speaking to!
Retrofit 2 corrects this mistake by only speaking bytes in its core.
You’re forced to bring a serialization format converter and configure it on each instance, even if it’s always JSON and Gson (please stop using Gson).</p>
<p>You can usually spot these types of problems in libraries because they start to accumulate weird
ceremony in order to support different use-cases like testing<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup>. It can be tempting as a library
author to over-correct away from exposing verbosity. By removing required configuration options and
reducing the use of inversion of control you make the happy path happy, but alternative use-cases
and alternative integrations become much harder.</p>
<p>Instead of trying to push verbosity down into the library when faced with situations like the view
binding activity usage above, package it into an integration library that’s easy to evolve or throw
away. When one of those integrations inevitably disappears, or a new one arrives, your core library
won’t need to change.</p>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>Um, this sentence is somewhat ridiculous, right? Testing is not a <em>different</em> use case. It’s a primary use case! I hope libraries come to mind here. Many do for me when I wrote it. <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
AssistedInject is dead, long live AssistedInject!2021-03-31T00:00:00+00:00https://jakewharton.com/assisted-inject-is-dead-long-live-assisted-injectThis post was published externally on Cash App Code Blog. Read it at https://code.cash.app/assisted-inject-is-dead-long-live-assisted-inject.A Jetpack Compose by any other name2020-12-30T00:00:00+00:00https://jakewharton.com/a-jetpack-compose-by-any-other-name<p>I really like Jetpack Compose.
Between work and personal stuff I have three projects which are each built on top of it.
It’s great!</p>
<p>So far my biggest problem is its name… but that requires some explaining.
Welcome to one of the hills I’ll die on!</p>
<h3 id="what-is-jetpack-compose">What is Jetpack Compose?</h3>
<p>If you’re already familiar with it, something should pop in your head when asked:
<em>What</em> is Jetpack Compose?</p>
<p>A new UI toolkit for Android? Yep, that’s right.
A declarative Android UI framework? Sure, that is correct.
A multiplatform application UI? Thanks to JetBrains this is also true.</p>
<p>If you’re somewhat in tune to how the sausage is made you may also reference the fact that it’s a
Kotlin compiler plugin and DSL to build Android UI or multiplatform UI. It’s those things, too.</p>
<p>None of these answers are wrong. However, they’re doing a bit of a disservice
to the internals of Compose and its unrealized potential.</p>
<h3 id="pedigree">Pedigree</h3>
<p>What we now know as Jetpack Compose started as two separate projects:</p>
<p>The first was a solution for writing declarative Android UIs using the existing platform UI toolkit.
Take the declarative components of React, wrap it in an ahead-of-time compiler like Svelte, and
target Android’s UI toolkit with Kotlin. All existing <code class="highlighter-rouge">View</code>-based UI could suddenly level-up by
changing their programming paradigm from imperative to declarative.</p>
<p>Separately, the toolkit team was about to ramp up on unbundling as many UI widgets as possible from
the OS. This followed on the success of <code class="highlighter-rouge">ViewPager</code>, <code class="highlighter-rouge">RecyclerView</code>, and what was learned from
the AppCompat and Design libraries. By removing a ton of OEM touchpoints and normalizing behavior
across all versions of Android, the work required to build a good UI would be reduced.</p>
<p>Over time these efforts became inescapably linked.</p>
<p>If you are building standalone versions of the platform UI widgets then why not take the opportunity
to correct mistakes in their API and overcome limitations of the resource system? And if you’re
changing their API, why not have the new declarative system target only these unbundled widgets?
Each project only empowered the other as they spiraled closer together.</p>
<p>In hindsight, it seems inevitable they would become a single effort. Being a single effort does
not necessarily mean tight coupling, however.</p>
<h3 id="layering">Layering</h3>
<p>Each of my three projects built on Compose do not use the new Compose UI toolkit. That can be a
confusing statement even to those who have done a lot of Compose work. Didn’t we just call them
inescapably linked? Didn’t we define it earlier as a UI toolkit?</p>
<p>While Compose became a single effort started from two projects, a layering and responsibility
split similar to those original projects still exists. In fact, that separation has only become
more defined.</p>
<p>What this means is that Compose is, at its core, a general-purpose tool for managing a tree of
nodes of any type. Well a “tree of nodes” describes just about anything, and as a result Compose can
target just about anything.</p>
<p>Those tree nodes could be the new Compose UI toolkit internals. But it could just as easily be the
old <code class="highlighter-rouge">View</code> nodes inside a <code class="highlighter-rouge">ViewGroup</code> tree, it could be a <code class="highlighter-rouge">RemoteView</code> and the various remote views
within its tree, or a notification and the content inside it.</p>
<p>The nodes don’t have to be UI related at all. The tree could exist at the presenter layer as view
state objects, at the data layer as model objects, or simply be a value tree of pure data.</p>
<p>Compose doesn’t care! This is the part I really like. This is the part that’s great.</p>
<p>Separately, however, Compose is <em>also</em> a new UI toolkit and DSL which renders applications on
Android and Desktop. This part of Compose is built on top of the aforementioned core as a tree of
nodes which happen to be able to render themselves to a canvas.</p>
<p>The two parts are called the Compose compiler/runtime and Compose UI, respectively.</p>
<p>This separation of concerns is very welcome. Conflating both under the name of Compose,
in my opinion, is not welcome.</p>
<h3 id="naming">Naming</h3>
<p>Adjusting the naming here would address two problems: specificity and pigeonholing.</p>
<p>Placing a general-purpose compiler and runtime with a specific UI toolkit implementation under an
umbrella name means discussions about them are imprecise by nature. This post started by saying
that I’m working on three projects built on Compose… did you think I meant Compose UI?
You almost certainly did.</p>
<p>The Compose name is more akin to Jetpack than it is AppCompat. We don’t treat it like that, and
there are no signs of Google correcting our perception. So now I must endlessly clarify that I’m
working on three projects built on the Compose compiler/runtime which do not use the Compose UI
toolkit and oh, by the way, yes, those are separate things.</p>
<p>Maybe you don’t think this is a good enough reason to have two names. After all, how many people
are going to build something on just the compiler and runtime?</p>
<p>Yet that is all the <em>more</em> reason to rename it! You’ve just pigeonholed the project to not be
anything more than what it already is–a cool technology on which Compose UI is built. The possible
applications of the general-purpose Compose compiler/runtime are widespread and should be
encouraged. Right now it feels like Google buried the lede.</p>
<p>A separate name is an easy way to draw attention to the great work which is the compiler and runtime
of Compose. There’s been the rare tweet about it, the casual mention in a talk, and the occasional
blog post showcasing a different use, but aside from that there’s not much breathing room. The
excitement around Compose UI (which is also much deserved) drowns it out.</p>
<p>Compose compiler/runtime supports more platforms and targets than Compose UI. In addition to Android
I’ve run Compose-based projects on the JVM (in a server, not on desktop) and limped one along in
a non-browser JS engine. These are places where Compose UI is impossible, but Compose is not!</p>
<p>I’m excited for these three projects of mine to make their way into open source to showcase what the
Compose compiler and runtime can do on their own. I am not excited about having to continually
clarify that the Compose compiler/runtime duo are not related to Compose UI or to Android.</p>
<h3 id="crane">Crane</h3>
<p>The internal codename for the new UI toolkit was “crane”. Before it was public I voiced support
of retaining that name. After Compose was public I voiced support of using two names and using
“crane” for Compose UI. But messages in chat rooms are easy to ignore–even if some agreed.</p>
<p>Unfortunately, much time has passed, and the tea leaves are showing that Compose is about to enter
beta. It’s too late to make this naming change for Compose UI. No one even calls it Compose UI.
It’s always been just Compose.</p>
<p>So this blog post is a hail-mary plea to Google:
please rename the compiler and runtime to something else!</p>
<p>Compose is such a bland name anyway.
Name it Evergreen (like the trees).
Name it Juliet (who wrote the blog title).
Hell, name it Crane (for maximum internal confusion).
Give it the different name that it deserves so it can stand on its own.</p>
<p>But please, don’t relegate this amazing, general-purpose, multiplatform compiler and runtime to live
behind the blanket nomenclature that is just Compose!</p>
Treating Dockerfiles as shell scripts2020-12-03T00:00:00+00:00https://jakewharton.com/treating-dockerfiles-as-shell-scripts<p>I use Docker to run a lot of tools. With the tools all wrapped up in containers, my computers are
free of Python and Go and the various other dependencies needed for their use. While this is a nice
win for isolation and reproducibility, the user experience is sub-par.</p>
<p>To run a tool, I just use <code class="highlighter-rouge">docker run</code>:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ docker run --rm tool arguments...
</code></pre></div></div>
<p>But the editing workflow is something like:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ nano tool.dockerfile
# hack hack hack...
$ docker build -t tool - < tool.dockerfile
$ docker run --rm tool arguments...
</code></pre></div></div>
<p>I also use a lot of bash scripts with easy-to-remember names. To run a script I just type its name:
<code class="highlighter-rouge">./update_containers.sh</code>. To edit, I open it in an editor, save, and then run. The user experience
of this is top-notch!</p>
<p>Can we combine the two?</p>
<h3 id="executable-dockerfiles">Executable Dockerfiles</h3>
<p>If the first line of an executable starts with <code class="highlighter-rouge">#!</code>, unix-y systems will treat what follows on that
line as an executable for interpreting the rest of the file. This is called the
<a href="https://en.wikipedia.org/wiki/Shebang_(Unix)">shebang</a>, and the bash scripts I use start with one:
<code class="highlighter-rouge"><span class="c">#!/usr/bin/env bash</span></code>.</p>
<p>In order to do this with a Dockerfile, though, we need a program which will conditionally run
<code class="highlighter-rouge">docker build</code> and then <code class="highlighter-rouge">docker run</code> the resulting image. Thankfully <code class="highlighter-rouge">docker build</code> is already
conditional and won’t rebuild anything unless necessary, so we can always run it.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/usr/bin/env bash</span>
<span class="nv">NAME</span><span class="o">=</span><span class="si">$(</span><span class="nb">basename</span> <span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span><span class="si">)</span>
docker build <span class="nt">-t</span> <span class="s2">"</span><span class="nv">$NAME</span><span class="s2">"</span> - < <span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span> <span class="o">></span> /dev/null
<span class="nb">shift</span> <span class="c"># Remove script name from arguments</span>
docker run <span class="nt">--rm</span> <span class="nt">--name</span> <span class="s2">"</span><span class="nv">$NAME</span><span class="s2">"</span> <span class="s2">"</span><span class="nv">$NAME</span><span class="s2">"</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
</code></pre></div></div>
<p>With this saved as <code class="highlighter-rouge">dockerfile-shebang.sh</code>, we can add it as the shebang in a Dockerfile.</p>
<div class="language-docker highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/path/to/dockerfile-shebang.sh</span>
<span class="k">FROM</span><span class="s"> alpine:latest</span>
<span class="k">ENTRYPOINT</span><span class="s"> ["echo"]</span>
</code></pre></div></div>
<p>Saving this as <code class="highlighter-rouge">echo.dockerfile</code> and running <code class="highlighter-rouge">chmod +x echo.dockerfile</code> provides the user experience
we’re after:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ ./echo.dockerfile Hello, world!
Hello, world!
</code></pre></div></div>
<h3 id="its-dockerfile-shebang">It’s Dockerfile-Shebang!</h3>
<p>I have wrapped up this utility into an executable, <code class="highlighter-rouge">dockerfile-shabang</code>.
You can find it at
<a href="https://github.com/JakeWharton/dockerfile-shebang">github.com/JakeWharton/dockerfile-shebang</a>.</p>
<p>The implementation is a bit more complicated than above for a few usability and correctness
concerns:</p>
<ol>
<li>Builds can be slow, so a message will be displayed if the container is currently being built.</li>
<li>If the build step fails, its entire output will be displayed to aid in debugging.</li>
<li>Most importantly, there’s a mechanism for passing arguments to the <code class="highlighter-rouge">docker run</code> command for
mounting volumes, setting environment variables, and any other container-level flags.</li>
</ol>
<p>A real-world invocation looks something like:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ ./tool.dockerfile -v /tanker/backups:/backups -e UID=1000 -- /backups/path/to/file.txt
</code></pre></div></div>
<p>While I’ve been using Docker to wrap tools for a while, I’ve only been using this shebang for a
week. If you’re feeling similar usability pain around Dockerfiles, try it out, let me know if it
works, and let me know of any use cases you have which aren’t covered.</p>
Peeking at command-line ANSI escape sequences2020-10-28T00:00:00+00:00https://jakewharton.com/peeking-at-colorful-command-line-output<p>Command-line programs use color to convey additional information and to look pretty. For example, compare the output of <code class="highlighter-rouge">ls</code> with and without the <code class="highlighter-rouge">--color</code> flag:</p>
<p><a href="/static/post-image/ansi-ls.svg"><img src="/static/post-image/ansi-ls.svg" alt="The output of 'ls' and 'ls --color' in a folder with three entries, the latter command using color to distinguish an executable and folder from a regular file." /></a></p>
<p>The color helps convey information in this compact output that would otherwise only be available in more verbose forms (<code class="highlighter-rouge">-l</code>).</p>
<p>In addition to color, a program may update existing output. You can see this when updating images with <code class="highlighter-rouge">docker-compose</code>:</p>
<p><a href="/static/post-image/ansi-docker.svg"><img src="/static/post-image/ansi-docker.svg" alt="The output of 'docker-compose' showing three lines of progress bars updating individually." /></a></p>
<p>Both of these effects are created using something called ANSI escape sequences.</p>
<h3 id="ansi-escape-crash-course">ANSI escape crash course</h3>
<p>Reading <a href="https://en.wikipedia.org/wiki/ANSI_escape_code">the Wikipedia entry on ANSI escapes</a> is a great starting point for learning how to recreate these examples. Each escape sequence starts with a 0x1B (escape) character followed usually by <code class="highlighter-rouge">[</code> and then one or more commands using letters or numbers.</p>
<p>The <code class="highlighter-rouge">ls</code> example above uses green and blue text as well as making the colored entries bold which we can recreate.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>echo -e "\e[1;32mbinary\e[0m file \e[1;34mfolder\e[0m"
</code></pre></div></div>
<p><a href="/static/post-image/ansi-ls-echo.svg"><img src="/static/post-image/ansi-ls-echo.svg" alt="The output of the 'echo' command replicating the output of 'ls'." /></a></p>
<p>Let’s break down the interesting parts:</p>
<ul>
<li><code class="highlighter-rouge">echo -e</code> – Adding the <code class="highlighter-rouge">-e</code> flag to <code class="highlighter-rouge">echo</code> instructs it to enable backslash escapes.</li>
<li><code class="highlighter-rouge">\e[1;32m</code> – <code class="highlighter-rouge">\e</code> is a backslash escape for the 0x1B escape character and the <code class="highlighter-rouge">[</code> starts a sequence. <code class="highlighter-rouge">1</code> enables bold and <code class="highlighter-rouge">32</code> is the color green. Numbers are separated by <code class="highlighter-rouge">;</code> and terminated by <code class="highlighter-rouge">m</code>. Anything that follows will now be displayed as bold and green.</li>
<li><code class="highlighter-rouge">\e[0m</code> – Once again <code class="highlighter-rouge">\e[</code> starts a sequence and <code class="highlighter-rouge">m</code> terminates it. The <code class="highlighter-rouge">0</code> clears all previous formatting.</li>
<li><code class="highlighter-rouge">\e[1;34m</code> – Nearly identical to the sequence from before except it uses <code class="highlighter-rouge">34</code> for a blue color.</li>
</ul>
<p>The <code class="highlighter-rouge">docker-compose</code> example moves the cursor to rewrite previous output which we can begin to recreate.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>echo "Pulling zulu-jdk-15 ... downloading" && \
echo "Pulling zulu-jdk-11 ... downloading" && \
echo "Pulling zulu-jdk-8 ... downloading" && \
sleep 2 && \
echo -e "\e[2A\e[24C\e[32mdone\e[0m\e[K" && \
sleep 1 && \
echo -e "\e[24C\e[32mdone\e[0m\e[K" && \
sleep 1 && \
echo -e "\e[3A\e[24C\e[32mdone\e[0m\e[K\n\n"
</code></pre></div></div>
<p><a href="/static/post-image/ansi-docker-echo.svg"><img src="/static/post-image/ansi-docker-echo.svg" alt="The output of the 'echo' commands replicating the output of 'docker-compose'." /></a></p>
<p>Let’s break down the interesting parts for this example:</p>
<ul>
<li><code class="highlighter-rouge">\e[2A</code> – Each <code class="highlighter-rouge">echo</code> emits a trailing newline, so after the third <code class="highlighter-rouge">echo</code> our cursor is below the third line at column 0. This command moves the cursor up (<code class="highlighter-rouge">A</code>) by two lines placing it on the “zulu-jdk-11” line still at column 0.</li>
<li><code class="highlighter-rouge">\e[24C</code> – Move the cursor to the right (<code class="highlighter-rouge">C</code>) by 24 columns. This places the cursor directly before the “d” in “downloading”.</li>
<li><code class="highlighter-rouge">\e[32m</code> – Set the color to green. Remember this from the last section?</li>
<li><code class="highlighter-rouge">\e[K</code> – After writing “done”, the “loading” part of “downloading” is still visible. This command clears the current line from the cursor position to the line end.</li>
</ul>
<p>With these ANSI escape sequences we can recreate existing programs and being to create our own. But how do we know whether we’re using the same techniques as these programs? And if we don’t know how to produce a particular output how can we discover how it was created?</p>
<h3 id="displaying-ansi-sequences">Displaying ANSI sequences</h3>
<p>Given that ANSI sequences start with the 0x1B character and then <code class="highlighter-rouge">[</code> we can replace that escape with something else to disable it.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ls --color | sed -r 's/\x1b\[/\\e\[/g'
</code></pre></div></div>
<p><a href="/static/post-image/ansi-ls-sed.svg"><img src="/static/post-image/ansi-ls-sed.svg" alt="The output of 'ls' piped through 'sed' replacing ANSI escapes with printable characters" /></a></p>
<p>The <code class="highlighter-rouge">sed</code> command<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup> matches 0x1B and <code class="highlighter-rouge">[</code> and replaces it with <code class="highlighter-rouge">\e[</code> which is shown as normal text. This particular replacement is convenient because you can copy the output into an <code class="highlighter-rouge">echo</code> and see the rendered form.</p>
<p>In this output we can see <code class="highlighter-rouge">ls</code> is using almost exactly the same ANSI sequences as we were. The only addition is that they start with <code class="highlighter-rouge">\e[0m</code> in order to clear any existing formatting.</p>
<p>You may also notice that the output has changed to list each entry on its own line rather than on a single line. This is because <code class="highlighter-rouge">ls</code> detects that its output is going into a pipe rather than to a terminal display. Programs may also choose to omit color when piped which defeats the whole purpose of adding the <code class="highlighter-rouge">sed</code> command. To solve both cases, run the program using <code class="highlighter-rouge">unbuffer</code> before piping.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>unbuffer ls --color | sed -r 's/\x1b\[/\\e\[/g'
</code></pre></div></div>
<p><a href="/static/post-image/ansi-ls-sed-unbuffer.svg"><img src="/static/post-image/ansi-ls-sed-unbuffer.svg" alt="The output of 'ls' with 'unbuffer' piped through 'sed'" /></a></p>
<p>With the pipe usage hidden by <code class="highlighter-rouge">unbuffer</code>, the output of <code class="highlighter-rouge">ls</code> is back to being a single line.</p>
<p>If you run <code class="highlighter-rouge">docker-compose</code> with <code class="highlighter-rouge">unbuffer</code> and piping to <code class="highlighter-rouge">sed</code> the result is clearly not correct:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>unbuffer docker-compose pull | sed -r 's/\x1b\[/\\e\[/g'
</code></pre></div></div>
<p><a href="/static/post-image/ansi-docker-sed.svg"><img src="/static/post-image/ansi-docker-sed.svg" alt="The output of 'docker-compose' with 'unbuffer' piped through 'sed'" /></a></p>
<p>This is because <code class="highlighter-rouge">docker-compose</code> is using carriage returns (<code class="highlighter-rouge">\r</code>) to move the cursor back to column 0 on a line. We can update our <code class="highlighter-rouge">sed</code> to include a command to escape carriage returns too.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>unbuffer docker-compose pull | sed -r -e 's/\x0d/\\r/g' -e 's/\x1b\[/\\e\[/g'
</code></pre></div></div>
<p><a href="/static/post-image/ansi-docker-sed-2.svg"><img src="/static/post-image/ansi-docker-sed-2.svg" alt="The output of 'docker-compose' now with carriage return escaping" /></a></p>
<p>Now we can see all the commands. There is a lot of output here because <code class="highlighter-rouge">docker-compose</code> is updating the display very rapidly. Unlike our toy version above, each line is fully rewritten for each update. At the very end, though, you can see the <code class="highlighter-rouge">\e[32mdone\e[0m</code> sequence as part of updating the “zulu-jdk-15” line.</p>
<h3 id="bonus-technique-asciinema">Bonus technique: Asciinema</h3>
<p><a href="https://asciinema.org/">Asciinema</a> can also be used to inspect ANSI sequences, carriage returns, and everything else that a program outputs. Every terminal image and animation captured in this post was captured using Asciinema before being fed to <a href="https://github.com/marionebl/svg-term-cli"><code class="highlighter-rouge">svg-term</code></a>.</p>
<p>For example, the <code class="highlighter-rouge">docker-compose</code> output can be captured like this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>asciinema rec -c "docker-compose pull" docker.json
</code></pre></div></div>
<p><a href="/static/post-image/ansi-asciinema.svg"><img src="/static/post-image/ansi-asciinema.svg" alt="Using 'asciinema' to capture the output of the 'docker-compose' command" /></a></p>
<p><em>(Yes, I captured the above example of using Asciinema inside Asciinema!)</em></p>
<p>The resulting <code class="highlighter-rouge">docker.json</code> contains a series of JSON objects which describe the output commands.</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="nl">"version"</span><span class="p">:</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w"> </span><span class="nl">"width"</span><span class="p">:</span><span class="w"> </span><span class="mi">122</span><span class="p">,</span><span class="w"> </span><span class="nl">"height"</span><span class="p">:</span><span class="w"> </span><span class="mi">48</span><span class="p">,</span><span class="w"> </span><span class="nl">"timestamp"</span><span class="p">:</span><span class="w"> </span><span class="mi">1603858671</span><span class="p">,</span><span class="w"> </span><span class="nl">"env"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="nl">"SHELL"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/bin/bash"</span><span class="p">,</span><span class="w"> </span><span class="nl">"TERM"</span><span class="p">:</span><span class="w"> </span><span class="s2">"xterm-256color"</span><span class="p">}}</span><span class="w">
</span><span class="p">[</span><span class="mf">0.412745</span><span class="p">,</span><span class="w"> </span><span class="s2">"o"</span><span class="p">,</span><span class="w"> </span><span class="s2">"Pulling zulu-jdk-15 ... </span><span class="se">\r\r\n</span><span class="s2">Pulling zulu-jdk-11 ... </span><span class="se">\r\r\n</span><span class="s2">Pulling zulu-jdk-8 ... </span><span class="se">\r\r\n</span><span class="s2">"</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="mf">0.671883</span><span class="p">,</span><span class="w"> </span><span class="s2">"o"</span><span class="p">,</span><span class="w"> </span><span class="s2">"</span><span class="se">\u</span><span class="s2">001b[1A</span><span class="se">\u</span><span class="s2">001b[2K</span><span class="se">\r</span><span class="s2">Pulling zulu-jdk-8 ... pulling from azul/zulu-openjdk</span><span class="se">\r\u</span><span class="s2">001b[1B"</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="mf">0.672048</span><span class="p">,</span><span class="w"> </span><span class="s2">"o"</span><span class="p">,</span><span class="w"> </span><span class="s2">"</span><span class="se">\u</span><span class="s2">001b[1A</span><span class="se">\u</span><span class="s2">001b[2K</span><span class="se">\r</span><span class="s2">Pulling zulu-jdk-8 ... digest: sha256:13d16ca0335fbe1df3...</span><span class="se">\r\u</span><span class="s2">001b[1B"</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="mf">0.672159</span><span class="p">,</span><span class="w"> </span><span class="s2">"o"</span><span class="p">,</span><span class="w"> </span><span class="s2">"</span><span class="se">\u</span><span class="s2">001b[1A</span><span class="se">\u</span><span class="s2">001b[2K</span><span class="se">\r</span><span class="s2">Pulling zulu-jdk-8 ... status: image is up to date for a...</span><span class="se">\r\u</span><span class="s2">001b[1B"</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="mf">0.672478</span><span class="p">,</span><span class="w"> </span><span class="s2">"o"</span><span class="p">,</span><span class="w"> </span><span class="s2">"</span><span class="se">\u</span><span class="s2">001b[1A</span><span class="se">\u</span><span class="s2">001b[2K</span><span class="se">\r</span><span class="s2">"</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="mf">0.672507</span><span class="p">,</span><span class="w"> </span><span class="s2">"o"</span><span class="p">,</span><span class="w"> </span><span class="s2">"Pulling zulu-jdk-8 ... </span><span class="se">\u</span><span class="s2">001b[32mdone</span><span class="se">\u</span><span class="s2">001b[0m</span><span class="se">\r\u</span><span class="s2">001b[1B"</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="mf">0.782864</span><span class="p">,</span><span class="w"> </span><span class="s2">"o"</span><span class="p">,</span><span class="w"> </span><span class="s2">"</span><span class="se">\u</span><span class="s2">001b[2A</span><span class="se">\u</span><span class="s2">001b[2K</span><span class="se">\r</span><span class="s2">Pulling zulu-jdk-11 ... pulling from azul/zulu-openjdk</span><span class="se">\r\u</span><span class="s2">001b[2B"</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="mf">0.782985</span><span class="p">,</span><span class="w"> </span><span class="s2">"o"</span><span class="p">,</span><span class="w"> </span><span class="s2">"</span><span class="se">\u</span><span class="s2">001b[2A</span><span class="se">\u</span><span class="s2">001b[2K</span><span class="se">\r</span><span class="s2">"</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="mf">0.78307</span><span class="p">,</span><span class="w"> </span><span class="s2">"o"</span><span class="p">,</span><span class="w"> </span><span class="s2">"Pulling zulu-jdk-11 ... digest: sha256:315e0a2a7b6bcc2343...</span><span class="se">\r\u</span><span class="s2">001b[2B"</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="mf">0.783146</span><span class="p">,</span><span class="w"> </span><span class="s2">"o"</span><span class="p">,</span><span class="w"> </span><span class="s2">"</span><span class="se">\u</span><span class="s2">001b[2A</span><span class="se">\u</span><span class="s2">001b[2K</span><span class="se">\r</span><span class="s2">Pulling zulu-jdk-11 ... status: image is up to date for a...</span><span class="se">\r\u</span><span class="s2">001b[2B"</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="mf">0.783372</span><span class="p">,</span><span class="w"> </span><span class="s2">"o"</span><span class="p">,</span><span class="w"> </span><span class="s2">"</span><span class="se">\u</span><span class="s2">001b[2A</span><span class="se">\u</span><span class="s2">001b[2K</span><span class="se">\r</span><span class="s2">"</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="mf">0.783428</span><span class="p">,</span><span class="w"> </span><span class="s2">"o"</span><span class="p">,</span><span class="w"> </span><span class="s2">"Pulling zulu-jdk-11 ... </span><span class="se">\u</span><span class="s2">001b[32mdone</span><span class="se">\u</span><span class="s2">001b[0m</span><span class="se">\r\u</span><span class="s2">001b[2B"</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="mf">1.091186</span><span class="p">,</span><span class="w"> </span><span class="s2">"o"</span><span class="p">,</span><span class="w"> </span><span class="s2">"</span><span class="se">\u</span><span class="s2">001b[3A</span><span class="se">\u</span><span class="s2">001b[2K</span><span class="se">\r</span><span class="s2">Pulling zulu-jdk-15 ... pulling from azul/zulu-openjdk</span><span class="se">\r\u</span><span class="s2">001b[3B"</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="mf">1.09136</span><span class="p">,</span><span class="w"> </span><span class="s2">"o"</span><span class="p">,</span><span class="w"> </span><span class="s2">"</span><span class="se">\u</span><span class="s2">001b[3A</span><span class="se">\u</span><span class="s2">001b[2K</span><span class="se">\r</span><span class="s2">Pulling zulu-jdk-15 ... digest: sha256:bf2d25e46d2c9fc373...</span><span class="se">\r\u</span><span class="s2">001b[3B"</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="mf">1.091511</span><span class="p">,</span><span class="w"> </span><span class="s2">"o"</span><span class="p">,</span><span class="w"> </span><span class="s2">"</span><span class="se">\u</span><span class="s2">001b[3A</span><span class="se">\u</span><span class="s2">001b[2K</span><span class="se">\r</span><span class="s2">"</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="mf">1.091571</span><span class="p">,</span><span class="w"> </span><span class="s2">"o"</span><span class="p">,</span><span class="w"> </span><span class="s2">"Pulling zulu-jdk-15 ... status: image is up to date for a...</span><span class="se">\r\u</span><span class="s2">001b[3B"</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="mf">1.091859</span><span class="p">,</span><span class="w"> </span><span class="s2">"o"</span><span class="p">,</span><span class="w"> </span><span class="s2">"</span><span class="se">\u</span><span class="s2">001b[3A</span><span class="se">\u</span><span class="s2">001b[2K</span><span class="se">\r</span><span class="s2">Pulling zulu-jdk-15 ... </span><span class="se">\u</span><span class="s2">001b[32mdone</span><span class="se">\u</span><span class="s2">001b[0m</span><span class="se">\r</span><span class="s2">"</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="mf">1.091919</span><span class="p">,</span><span class="w"> </span><span class="s2">"o"</span><span class="p">,</span><span class="w"> </span><span class="s2">"</span><span class="se">\u</span><span class="s2">001b[3B"</span><span class="p">]</span><span class="w">
</span></code></pre></div></div>
<p>For a complex output like <code class="highlighter-rouge">docker-compose</code> the JSON form can be easier to understand. One other advantage is that each individual write to standard out gets its own line whereas with the <code class="highlighter-rouge">sed</code> escape technique we don’t differentiate individual writes.</p>
<hr />
<p>If you use tools like Docker, Gradle, Bazel, and even just <code class="highlighter-rouge">ls</code> you may be familiar with seeing colored and updating output daily. By using tools like <code class="highlighter-rouge">sed</code> and <code class="highlighter-rouge">asciinema</code> you can learn how those tools render their output. Should you find yourself building a command-line tool in the future, knowledge of how to use these ANSI sequences can help delight your users–even if it’s only yourself!</p>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>If you are on Mac OS, you’ll need GNU <code class="highlighter-rouge">sed</code> for the <code class="highlighter-rouge">-r</code> flag which can be installed via <code class="highlighter-rouge">brew install gnu-sed</code> and then used as <code class="highlighter-rouge">gsed</code> or by <code class="highlighter-rouge">alias sed=gsed</code>. <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
Smaller APKs with resource optimization2020-09-01T00:00:00+00:00https://jakewharton.com/smaller-apks-with-resource-optimization<p>How many times does the name of a layout file appear in an Android APK? We can build a minimal APK with a single layout file to count the occurrences empirically.</p>
<p>Building an Android app with Gradle requires only one thing: an <code class="highlighter-rouge">AndroidManifest.xml</code> file with a package. From there we can add a dummy layout whose contents are just <code class="highlighter-rouge"><merge/></code> since we only care about its name.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>.
├── build.gradle
└── src
└── main
├── AndroidManifest.xml
└── res
└── layout
└── home_view.xml
</code></pre></div></div>
<p>Running <code class="highlighter-rouge">gradle assembleRelease</code> will produce a release APK measuring a paltry 2,118 bytes. We can dump its contents using <code class="highlighter-rouge">xxd</code> and look for <code class="highlighter-rouge">home_view</code> byte sequences.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ xxd build/outputs/apk/release/app-release-unsigned.apk
⋮
000004c0: 0000 0074 0000 0018 0000 0072 6573 2f6c ...t.......res/l
000004d0: 6179 6f75 742f 686f 6d65 5f76 6965 772e ayout/home_view.
000004e0: 786d 6c63 66e0 6028 6160 6060 6490 61d0 xmlcf.`(a```d.a.
⋮
00000570: 0000 0000 0000 0000 1818 7265 732f 6c61 ..........res/la
00000580: 796f 7574 2f68 6f6d 655f 7669 6577 2e78 yout/home_view.x
00000590: 6d6c 0000 0002 2001 f801 0000 7f00 0000 ml.... .........
⋮
00000700: 0000 0000 0909 686f 6d65 5f76 6965 7700 ......home_view.
00000710: 0202 1000 1400 0000 0100 0000 0100 0000 ................
⋮
00000870: 0000 ad04 0000 7265 732f 6c61 796f 7574 ......res/layout
00000880: 2f68 6f6d 655f 7669 6577 2e78 6d6c 504b /home_view.xmlPK
⋮
</code></pre></div></div>
<p>There are three uncompressed occurrences of the path and one uncompressed occurrence of only the name in the APK based on this output.</p>
<p>If you have not read my <a href="https://jakewharton.com/calculating-zip-file-entry-true-impact/">post on calculating zip entry size</a> or are not familiar with <a href="https://en.wikipedia.org/wiki/Zip_(file_format)#Structure">the structure of a zip file</a>, a zip file is a list of file entries followed by a directory of all available entries. Each entry contains the file path and so does the directory. This accounts for the first occurrence (the entry header) and the last occurrence (the directory record) in the output.</p>
<p>The middle two occurrences in the output are from inside the <code class="highlighter-rouge">resources.arsc</code> file which is a database of sorts for resources. Its contents are visible because the file is uncompressed inside the APK. Running <code class="highlighter-rouge">aapt dump --values resources build/outputs/apk/release/app-release-unsigned.apk</code> shows the <code class="highlighter-rouge">home_view</code> record and its mapping to the path:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Package Groups (1)
Package Group 0 id=0x7f packageCount=1 name=com.example
Package 0 id=0x7f name=com.example
type 0 configCount=1 entryCount=1
spec resource 0x7f010000 com.example:layout/home_view: flags=0x00000000
config (default):
resource 0x7f010000 com.example:layout/home_view: t=0x03 d=0x00000000 (s=0x0008 r=0x00)
(string8) "res/layout/home_view.xml"
</code></pre></div></div>
<p>The APK contains a fifth occurrence of the name inside the <code class="highlighter-rouge">classes.dex</code> file. It does not show up in the <code class="highlighter-rouge">xxd</code> output because the file is compressed. Running <code class="highlighter-rouge">baksmali dump <(unzip -p build/outputs/apk/release/app-release-unsigned.apk classes.dex)</code> shows the dex file’s string table which contains an entry for <code class="highlighter-rouge">home_view</code>:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> |[10] string_data_item
000227: 09 | utf16_size = 9
000228: 686f 6d65 5f76 6965| data = "home_view"
000230: 7700 |
</code></pre></div></div>
<p>This is for the field inside the <code class="highlighter-rouge">R.layout</code> class which maps the layout name to a unique integer value. Incidentally, that integer is the index into the <code class="highlighter-rouge">resources.arsc</code> database to look up the associated file name for reading its XML contents.</p>
<p>To summarize the answer to our question, for each resource file, the full path appears three times and the name appears twice.</p>
<h3 id="optimizing-resources">Optimizing resources</h3>
<p>Android Gradle plugin 4.2 introduces the <code class="highlighter-rouge">android.enableResourceOptimizations=true</code> flag which will run optimizations targeted for resources. This invokes the <code class="highlighter-rouge">aapt optimize</code> command on the merged resources and <code class="highlighter-rouge">resources.arsc</code> file before they are packaged into the APK. The optimization only applies to release builds and will run regardless of whether <code class="highlighter-rouge">minifyEnabled</code> is set to true.</p>
<p>With the flag added to <code class="highlighter-rouge">gradle.properties</code> we can compare two APKs using <a href="https://github.com/JakeWharton/diffuse">diffuse</a> to see its effects. The output is long, so we’ll break it apart by section.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> │ compressed │ uncompressed
├─────────┬───────┬───────┼─────────┬─────────┬───────
APK │ old │ new │ diff │ old │ new │ diff
──────────┼─────────┼───────┼───────┼─────────┼─────────┼───────
dex │ 695 B │ 695 B │ 0 B │ 1,016 B │ 1,016 B │ 0 B
arsc │ 682 B │ 674 B │ -8 B │ 576 B │ 564 B │ -12 B
manifest │ 535 B │ 535 B │ 0 B │ 1.1 KiB │ 1.1 KiB │ 0 B
res │ 185 B │ 157 B │ -28 B │ 116 B │ 116 B │ 0 B
asset │ 0 B │ 0 B │ 0 B │ 0 B │ 0 B │ 0 B
other │ 22 B │ 22 B │ 0 B │ 0 B │ 0 B │ 0 B
──────────┼─────────┼───────┼───────┼─────────┼─────────┼───────
total │ 2.1 KiB │ 2 KiB │ -36 B │ 2.7 KiB │ 2.7 KiB │ -12 B
</code></pre></div></div>
<p>First is a diff of the contents in the APK. The “compressed” columns are the size cost inside the APK, and the “uncompressed” columns are the cost when extracted.</p>
<p>The <code class="highlighter-rouge">res</code> category represents our single resource file whose size dropped 28 bytes. The <code class="highlighter-rouge">arsc</code> category is for the <code class="highlighter-rouge">resource.arsc</code> file which itself dropped 8 bytes. We’ll see the cause of these changes shortly.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> DEX │ old │ new │ diff
─────────┼─────┼─────┼───────────
files │ 1 │ 1 │ 0
strings │ 15 │ 15 │ 0 (+0 -0)
types │ 8 │ 8 │ 0 (+0 -0)
classes │ 2 │ 2 │ 0 (+0 -0)
methods │ 3 │ 3 │ 0 (+0 -0)
fields │ 1 │ 1 │ 0 (+0 -0)
ARSC │ old │ new │ diff
─────────┼─────┼─────┼──────
configs │ 1 │ 1 │ 0
entries │ 1 │ 1 │ 0
</code></pre></div></div>
<p>These two sections represent the code and contents of the resource database. Having no changes, we can infer that the optimizations have not affected the <code class="highlighter-rouge">R.layout.home_view</code> field nor the <code class="highlighter-rouge">home_view</code> resource entry.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>=================
==== APK ====
=================
compressed │ uncompressed │
───────┬────────┼───────┬────────┤
size │ diff │ size │ diff │ path
───────┼────────┼───────┼────────┼────────────────────────────
│ -185 B │ │ -116 B │ - res/layout/home_view.xml
157 B │ +157 B │ 116 B │ +116 B │ + res/eA.xml
674 B │ -8 B │ 564 B │ -12 B │ ∆ resources.arsc
───────┼────────┼───────┼────────┼────────────────────────────
831 B │ -36 B │ 680 B │ -12 B │ (total)
</code></pre></div></div>
<p>Finally, a granular diff of the file changes shows the effect of optimization. Our layout resource had its filename significantly truncated and was moved out of the <code class="highlighter-rouge">layout/</code> folder!</p>
<p>Inside the Gradle project, the folder and file names of XMLs have meaning. The folder is the resource type, and the name corresponds to the generated field and resource entry in the <code class="highlighter-rouge">.arsc</code> file. Once those files are inside the APK, however, the file path is meaningless and arbitrary. Resource optimization leverages this fact by making the names as short as possible<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup>.</p>
<p>The output of <code class="highlighter-rouge">aapt dump</code> confirms that the resource database also reflects the file change:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Package Groups (1)
Package Group 0 id=0x7f packageCount=1 name=com.example
Package 0 id=0x7f name=com.example
type 0 configCount=1 entryCount=1
spec resource 0x7f010000 com.example:layout/home_view: flags=0x00000000
config (default):
resource 0x7f010000 com.example:layout/home_view: t=0x03 d=0x00000000 (s=0x0008 r=0x00)
(string8) "res/eA.xml"
</code></pre></div></div>
<p>All three occurrences of the path in the APK are now shorter which results in the 36 byte savings. And while 36 bytes is a very small number, remember that the entire binary is only 2,118 bytes. A 36-byte savings is a 1.7% size reduction!</p>
<h3 id="real-world-examples">Real-world examples</h3>
<p>The resources of a real application number far more than just one. What does this optimization look like when applied to a real application?</p>
<h4 id="plaid">Plaid</h4>
<p>Nick Butcher’s <a href="https://github.com/android/plaid">Plaid</a> app has 734 resource files. In addition to their quantity, the names of the resource files are more descriptive (which is a fancy way of saying they’re longer). Instead of <code class="highlighter-rouge">home_view</code>, Plaid contains names like <code class="highlighter-rouge">searchback_stem_search_to_back.xml</code>, <code class="highlighter-rouge">attrs_elastic_drag_dismiss_frame_layout</code>, and <code class="highlighter-rouge">designer_news_story_description.xml</code>.</p>
<p>After updating the project to AGP 4.2, I used <code class="highlighter-rouge">diffuse</code> to compare a build without resource optimization to one with it enabled:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> │ compressed │ uncompressed
├───────────┬───────────┬───────────┼───────────┬───────────┬───────────
APK │ old │ new │ diff │ old │ new │ diff
──────────┼───────────┼───────────┼───────────┼───────────┼───────────┼───────────
dex │ 3.8 MiB │ 3.8 MiB │ 0 B │ 9.9 MiB │ 9.9 MiB │ 0 B
arsc │ 316.7 KiB │ 292.5 KiB │ -24.2 KiB │ 316.6 KiB │ 292.4 KiB │ -24.2 KiB
manifest │ 3 KiB │ 3 KiB │ 0 B │ 11.9 KiB │ 11.9 KiB │ 0 B
res │ 539.2 KiB │ 490.7 KiB │ -48.5 KiB │ 617.2 KiB │ 617.2 KiB │ 0 B
native │ 4.6 MiB │ 4.6 MiB │ 0 B │ 4.6 MiB │ 4.6 MiB │ 0 B
asset │ 0 B │ 0 B │ 0 B │ 0 B │ 0 B │ 0 B
other │ 83.6 KiB │ 83.6 KiB │ 0 B │ 128.6 KiB │ 128.6 KiB │ 0 B
──────────┼───────────┼───────────┼───────────┼───────────┼───────────┼───────────
total │ 9.4 MiB │ 9.3 MiB │ -72.7 KiB │ 15.6 MiB │ 15.5 MiB │ -24.2 KiB
</code></pre></div></div>
<p>Resource optimization netted a 0.76% savings on APK size. The native library size kept the impact smaller than I had hoped.</p>
<h4 id="seriesguide">SeriesGuide</h4>
<p>Uwe Trottmann’s <a href="https://github.com/UweTrottmann/SeriesGuide">SeriesGuide</a> app has 1044 resource files. Unlike Plaid, it is free of native libraries which should increase the impact of the optimization.</p>
<p>Once again I updated the project to AGP 4.2 and used <code class="highlighter-rouge">diffuse</code> to compare two builds:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> │ compressed │ uncompressed
├───────────┬───────────┬───────────┼───────────┬───────────┬───────────
APK │ old │ new │ diff │ old │ new │ diff
──────────┼───────────┼───────────┼───────────┼───────────┼───────────┼───────────
dex │ 2.4 MiB │ 2.4 MiB │ 0 B │ 5.7 MiB │ 5.7 MiB │ 0 B
arsc │ 1.7 MiB │ 1.6 MiB │ -32.9 KiB │ 1.7 MiB │ 1.6 MiB │ -32.9 KiB
manifest │ 5.6 KiB │ 5.6 KiB │ 0 B │ 28.3 KiB │ 28.3 KiB │ 0 B
res │ 693.9 KiB │ 628 KiB │ -66 KiB │ 992.2 KiB │ 992.2 KiB │ 0 B
asset │ 39.9 KiB │ 39.9 KiB │ 0 B │ 100.4 KiB │ 100.4 KiB │ 0 B
other │ 118.1 KiB │ 118.1 KiB │ 0 B │ 148.8 KiB │ 148.8 KiB │ 0 B
──────────┼───────────┼───────────┼───────────┼───────────┼───────────┼───────────
total │ 4.9 MiB │ 4.8 MiB │ -98.9 KiB │ 8.6 MiB │ 8.6 MiB │ -32.9 KiB
</code></pre></div></div>
<p>Here resource optimization was able to reduce the APK size by 2.0%!</p>
<h4 id="tivi">Tivi</h4>
<p>Chris Banes’ <a href="https://github.com/chrisbanes/tivi/">Tivi</a> app has a non-trivial subset written using <a href="https://developer.android.com/jetpack/compose">Jetpack Compose</a> which means fewer resources overall. A current build still contains 776 resource files.</p>
<p>By virtue of using Compose, Tivi is already using the latest AGP 4.2. With two quick builds we can see the impact of resource optimization:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> │ compressed │ uncompressed
├───────────┬───────────┬───────────┼───────────┬───────────┬───────────
APK │ old │ new │ diff │ old │ new │ diff
──────────┼───────────┼───────────┼───────────┼───────────┼───────────┼───────────
dex │ 3 MiB │ 3 MiB │ 0 B │ 6.8 MiB │ 6.8 MiB │ 0 B
arsc │ 363.4 KiB │ 337.9 KiB │ -25.6 KiB │ 363.3 KiB │ 337.7 KiB │ -25.6 KiB
manifest │ 3.6 KiB │ 3.6 KiB │ 0 B │ 16.1 KiB │ 16.1 KiB │ 0 B
res │ 680.4 KiB │ 629.2 KiB │ -51.2 KiB │ 1.2 MiB │ 1.2 MiB │ 0 B
asset │ 39.9 KiB │ 39.9 KiB │ 0 B │ 100.4 KiB │ 100.4 KiB │ 0 B
other │ 159.9 KiB │ 151.7 KiB │ -8.2 KiB │ 306.3 KiB │ 254.8 KiB │ -51.5 KiB
──────────┼───────────┼───────────┼───────────┼───────────┼───────────┼───────────
total │ 4.2 MiB │ 4.1 MiB │ -85 KiB │ 8.8 MiB │ 8.7 MiB │ -77.1 KiB
</code></pre></div></div>
<p>Once again we hit the 2.0% mark for APK size reduction!</p>
<h3 id="one-more-occurrence">One more occurrence</h3>
<p>All four examples so far have not used signed APKs. There are multiple versions of APK signing, and if your <code class="highlighter-rouge">minSdkVersion</code> is lower than 24 you are required include version 1 (V1) when signing. V1 signing uses <a href="https://docs.oracle.com/javase/tutorial/deployment/jar/intro.html">Java’s <code class="highlighter-rouge">.jar</code>
signing specification</a> which signs each file individually as a text entry in the <code class="highlighter-rouge">META-INF/MANIFEST.MF</code> file.</p>
<p>After creating and configuring a keystore for the original single-layout app, dumping the manifest file with <code class="highlighter-rouge">unzip -c build/outputs/apk/release/app-release.apk META-INF/MANIFEST.MF</code> shows these signatures:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Manifest-Version: 1.0
Built-By: Signflinger
Created-By: Android Gradle 4.2.0-alpha08
Name: AndroidManifest.xml
SHA-256-Digest: HdoGVd8U3Zjtf2VkGLExAPCQ1fq+kNL8eHKjVQXGI60=
Name: classes.dex
SHA-256-Digest: BVA1ApPvECg56DrrNPgD3jgv1edcM8VKYjcJEAG4G44=
Name: res/eA.xml
SHA-256-Digest: nDn7UQex2OWB3/AT054UvSAx9pYNSWwERCLfgdM6J6c=
Name: resources.arsc
SHA-256-Digest: 6w7i2Z9+LjwqlXS7YhhjzP/XhgvJF3PUuyJM60t0Qbw=
</code></pre></div></div>
<p>The full path of each file makes an appearance bringing the total occurrences of each resource path to four. Since shorter names will once again result in this file containing fewer bytes, resource optimization has an even greater impact.</p>
<hr />
<p>The Google-internal email which introduced me to this feature purported a savings of 1-3% on final APK size. Based on real-world tests this range seems to be about right. Ultimately the savings will depend on the size and number of resource files in your APK.</p>
<p>If you’re already using AGP 4.2 add <code class="highlighter-rouge">android.enableResourceOptimizations=true</code> to your <code class="highlighter-rouge">gradle.properties</code> and enjoy this free APK size savings. If you are not yet on AGP 4.2 add it anyway so that you don’t forget when you eventually upgrade!</p>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>In this example, notably, the name doesn’t seem as small as possible since it is two characters instead of one. A hash function computes the new name for each file. The number of resource files dictates the size of the hash which has a lower bound of two. The algorithm appears to work with a lower bound of one, so I’m not sure why the author chose to use two. Perhaps they didn’t expect projects to contain fewer than 64 resources. I sent <a href="https://r.android.com/1416749">r.android.com/1416749</a> to lower the bound. <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
Shrinking a Kotlin binary by 99.2%2020-08-24T00:00:00+00:00https://jakewharton.com/shrinking-a-kotlin-binary<p>We’ll get to the shrinking, but first let’s motivate the binary in question. Three years ago I wrote the <a href="https://developer.squareup.com/blog/surfacing-hidden-change-to-pull-requests/">“Surfacing Hidden Change to Pull Requests” post</a> which covered pushing important stats and diffs into PRs as a comment. This avoids surprises with changes that affect binary size, manifests, and dependency trees.</p>
<p>Showing dependency trees used Gradle’s <code class="highlighter-rouge">dependencies</code> task and <code class="highlighter-rouge">diff -U 0</code> to display changes from the previous commit. The example in that post bumped the Kotlin version from 1.1-M03 to 1.1-M04 producing the following diff:</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">@@ -125,2 +125,3 @@</span>
<span class="gd">-| \--- org.jetbrains.kotlin:kotlin-stdlib:1.0.4 -> 1.1-M03
-| \--- org.jetbrains.kotlin:kotlin-runtime:1.1-M03
</span><span class="gi">+| \--- org.jetbrains.kotlin:kotlin-stdlib:1.0.4 -> 1.1-M04
+| \--- org.jetbrains.kotlin:kotlin-runtime:1.1-M04
+| \--- org.jetbrains:annotations:13.0
</span><span class="p">@@ -145,2 +146 @@</span>
<span class="gd">-+--- org.jetbrains.kotlin:kotlin-stdlib:1.1-M03
-+--- org.jetbrains.kotlin:kotlin-runtime:1.1-M03
</span><span class="gi">++--- org.jetbrains.kotlin:kotlin-stdlib:1.1-M04
</span></code></pre></div></div>
<p>Aside from seeing the version bump reflected, there’s two extra facts here we can deduce about the change:</p>
<ol>
<li>The <code class="highlighter-rouge">kotlin-runtime</code> dependency gained a dependency on Jetbrains’ <code class="highlighter-rouge">annotations</code> artifact as seen in the first section of the diff.</li>
<li>A direct dependency on <code class="highlighter-rouge">kotlin-runtime</code> was removed as seen in the second section of the diff. This is fine, as the first section already tells us that <code class="highlighter-rouge">kotlin-runtime</code> is a dependency of <code class="highlighter-rouge">kotlin-stdlib</code>.</li>
</ol>
<p>These two facts are shown in the displayed diff, but there’s a subtle third fact which is only implied. Because the first section is indented, we know that one of our direct dependencies has a transitive dependency on <code class="highlighter-rouge">kotlin-stdlib</code>. Unfortunately we have no idea which dependency is affected.</p>
<p>To solve this problem I wrote a tool called <code class="highlighter-rouge">dependency-tree-diff</code> which shows the path to a root dependency for any changes in the tree.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> +--- com.jakewharton.rxbinding:rxbinding-kotlin:1.0.0
<span class="gd">-| \--- org.jetbrains.kotlin:kotlin-stdlib:1.0.4 -> 1.1-M03
-| \--- org.jetbrains.kotlin:kotlin-runtime:1.1-M03
</span><span class="gi">+| \--- org.jetbrains.kotlin:kotlin-stdlib:1.0.4 -> 1.1-M04
+| \--- org.jetbrains.kotlin:kotlin-runtime:1.1-M04
+| \--- org.jetbrains:annotations:13.0
</span><span class="gd">-+--- org.jetbrains.kotlin:kotlin-stdlib:1.1-M03 (*)
-\--- org.jetbrains.kotlin:kotlin-runtime:1.1-M03
</span><span class="gi">+\--- org.jetbrains.kotlin:kotlin-stdlib:1.1-M04 (*)
</span></code></pre></div></div>
<p>Our implicit third fact, which other direct dependency was affected, is now explicit in the output. Change authors can now reflect whether there may be any compatibility issues with the affected dependencies.</p>
<p>You can learn more about the tool and see another example <a href="https://github.com/JakeWharton/dependency-tree-diff">in its README</a>.</p>
<h3 id="shrinking-the-binary">Shrinking the binary</h3>
<p>This tool needs to be checked into our repo and run on CI. Having successfully built <a href="https://github.com/JakeWharton/adb-event-mirror/">adb-event-mirror</a> using Kotlin script the first version of this tool also used Kotlin script. While it worked and was tiny, <code class="highlighter-rouge">kotlinc</code> is not installed on the CI machines. We rely on the Kotlin Gradle plugin to compile Kotlin, not a standalone binary.</p>
<p>You can locally redirect the Kotlin script cache directory to capture the compiled jar, but it still depends on the Kotlin script artifact which is large, has lots of dependencies, and are still quite dynamic. It was clear this wasn’t the right path, but I filed <a href="https://youtrack.jetbrains.com/issue/KT-41304">KT-41304</a> to hopefully make producing a fat <code class="highlighter-rouge">.jar</code> of a script easier in the future.</p>
<p>I switched to a classic Kotlin Gradle project and produced a fat <code class="highlighter-rouge">.jar</code> with the <code class="highlighter-rouge">kotlin-stdlib</code> dependency included. After prepending a script to <a href="https://skife.org/java/unix/2011/06/20/really_executable_jars.html">make the jar self-executing</a>, the binary clocked in 1699978 bytes (or ~1.62MiB). Not bad, but we can do better!</p>
<h4 id="removing-kotlin-metadata">Removing Kotlin metadata</h4>
<p>Listing the files in the <code class="highlighter-rouge">.jar</code> using <code class="highlighter-rouge">unzip -l</code> shows that aside from <code class="highlighter-rouge">.class</code>, the majority are <code class="highlighter-rouge">.kotlin_module</code> or <code class="highlighter-rouge">.kotlin_metadata</code>. These are used by the Kotlin compiler and by Kotlin’s reflection and neither are needed for our binary.</p>
<p>We can filter these out of the binary along with <code class="highlighter-rouge">module-info.class</code> which is used for Java 9’s module system and files in <code class="highlighter-rouge">META-INF/maven/</code> which propagate information about projects built with the Maven tool.</p>
<p>Removing all these files yields a new binary size of 1513414 bytes (~1.44MiB), an 11% reduction in size.</p>
<h4 id="using-r8">Using R8</h4>
<p>R8 is the code optimizer and obfuscator for Android builds. While it’s normally used to optimize and obfuscate Java classfiles during conversion to the Dalvik executable format, it also supports outputting Java classfiles. In order to use it, we need to specify the entry point to the tool using ProGuard’s configuration syntax.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-dontobfuscate
-keepattributes SourceFile, LineNumberTable
-keep class com.jakewharton.gradle.dependencies.DependencyTreeDiff {
public static void main(java.lang.String[]);
}
</code></pre></div></div>
<p>In addition to the entrypoint, obfuscation is disabled, and we retain the source file and line number attributes so that any exceptions which occur will still be understandable.</p>
<p>Passing the fat <code class="highlighter-rouge">.jar</code> through R8 produces a new minified <code class="highlighter-rouge">.jar</code> which can then be made executable. The resulting binary is now just 41680 bytes (~41KiB), a 98% reduction in size. Nice!</p>
<p>Since we are producing a binary and not a library, the <code class="highlighter-rouge">-allowaccessmodification</code> option will make optimizations like class merging and inlining more effective by allowing hidden members to be made public. Adding this produces a binary of 37630 bytes (~37KiB).</p>
<h4 id="tweaking-standard-library-usage">Tweaking standard library usage</h4>
<p>It is absolutely safe to stop here, but I’m bad at stopping…</p>
<p>Now that the binary is sufficiently small we can start looking at what code is contributing to the size. Normally I would turn to <code class="highlighter-rouge">javap</code> for peeking at bytecode, but since we only care about seeing API calls we can unzip the binary and open the classfiles in IntelliJ IDEA which will use the Fernflower decompiler to show roughly-equivalent Java.</p>
<p>The <code class="highlighter-rouge">main</code> method starts by reading in the arguments as files:</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fun</span> <span class="nf">main</span><span class="p">(</span><span class="k">vararg</span> <span class="n">args</span><span class="p">:</span> <span class="nc">String</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">args</span><span class="p">.</span><span class="n">size</span> <span class="p">==</span> <span class="mi">2</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">val</span> <span class="py">old</span> <span class="p">=</span> <span class="n">args</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nf">let</span><span class="p">(</span><span class="o">::</span><span class="nc">File</span><span class="p">).</span><span class="nf">readText</span><span class="p">()</span>
<span class="kd">val</span> <span class="py">new</span> <span class="p">=</span> <span class="n">args</span><span class="p">[</span><span class="mi">1</span><span class="p">].</span><span class="nf">let</span><span class="p">(</span><span class="o">::</span><span class="nc">File</span><span class="p">).</span><span class="nf">readText</span><span class="p">()</span>
</code></pre></div></div>
<p>The decompiled code looks like this:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">...</span> <span class="n">var0</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">Intrinsics</span><span class="o">.</span><span class="na">checkNotNullParameter</span><span class="o">(</span><span class="n">var0</span><span class="o">,</span> <span class="s">"args"</span><span class="o">);</span>
<span class="k">if</span> <span class="o">(</span><span class="n">var0</span><span class="o">.</span><span class="na">length</span> <span class="o">==</span> <span class="mi">2</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">String</span><span class="o">[]</span> <span class="n">var10000</span> <span class="o">=</span> <span class="n">var0</span><span class="o">;</span>
<span class="nc">String</span> <span class="n">var3</span> <span class="o">=</span> <span class="n">var0</span><span class="o">[</span><span class="mi">0</span><span class="o">];</span>
<span class="n">var3</span> <span class="o">=</span> <span class="n">FilesKt__FileReadWriteKt</span><span class="o">.</span><span class="na">readText</span><span class="n">$default</span><span class="o">(</span><span class="k">new</span> <span class="nc">File</span><span class="o">(</span><span class="n">var3</span><span class="o">),</span> <span class="o">(</span><span class="nc">Charset</span><span class="o">)</span><span class="kc">null</span><span class="o">,</span> <span class="mi">1</span><span class="o">);</span>
<span class="nc">String</span> <span class="n">var1</span> <span class="o">=</span> <span class="n">var10000</span><span class="o">[</span><span class="mi">1</span><span class="o">];</span>
<span class="nc">String</span> <span class="n">var8</span> <span class="o">=</span> <span class="n">FilesKt__FileReadWriteKt</span><span class="o">.</span><span class="na">readText</span><span class="n">$default</span><span class="o">(</span><span class="k">new</span> <span class="nc">File</span><span class="o">(</span><span class="n">var1</span><span class="o">),</span> <span class="o">(</span><span class="nc">Charset</span><span class="o">)</span><span class="kc">null</span><span class="o">,</span> <span class="mi">1</span><span class="o">);</span>
</code></pre></div></div>
<p>Peeking at <code class="highlighter-rouge">FilesKt__FileReadWriteKt</code> shows the unfortunate file reading code we’ve all written at some point in the past, and it pulls in <code class="highlighter-rouge">kotlin.ExceptionsKt</code>, <code class="highlighter-rouge">kotlin.jvm.internal.Intrinsics</code>, and <code class="highlighter-rouge">kotlin.text.Charsets</code>.</p>
<p>Switching from <code class="highlighter-rouge">java.io.File</code> to <code class="highlighter-rouge">java.nio.path.Path</code> means we can use a built-in method for reading the contents.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> fun main(vararg args: String) {
if (args.size == 2) {
<span class="gd">- val old = args[0].let(::File).readText()
- val new = args[1].let(::File).readText()
</span><span class="gi">+ val old = args[0].let(Paths::get).let(Paths::readString)
+ val new = args[1].let(Paths::get).let(Paths::readString)
</span></code></pre></div></div>
<p>With these changes the binary drops to 30914 bytes (~30KiB).</p>
<p>Another standard library usage that caught my eye is splitting the inputs by line:</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">private</span> <span class="k">fun</span> <span class="nf">findDependencyPaths</span><span class="p">(</span><span class="n">text</span><span class="p">:</span> <span class="nc">String</span><span class="p">):</span> <span class="nc">Set</span><span class="p"><</span><span class="nc">List</span><span class="p"><</span><span class="nc">String</span><span class="p">>></span> <span class="p">{</span>
<span class="kd">val</span> <span class="py">dependencyLines</span> <span class="p">=</span> <span class="n">text</span><span class="p">.</span><span class="nf">lineSequence</span><span class="p">()</span>
<span class="p">.</span><span class="nf">dropWhile</span> <span class="p">{</span> <span class="p">!</span><span class="n">it</span><span class="p">.</span><span class="nf">startsWith</span><span class="p">(</span><span class="s">"+--- "</span><span class="p">)</span> <span class="p">}</span>
<span class="p">.</span><span class="nf">takeWhile</span> <span class="p">{</span> <span class="n">it</span><span class="p">.</span><span class="nf">isNotEmpty</span><span class="p">()</span> <span class="p">}</span>
</code></pre></div></div>
<p>The decompiled Java looks somewhat like this:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">Set</span> <span class="nf">findDependencyPaths</span><span class="o">(</span><span class="nc">String</span> <span class="n">var0</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">String</span><span class="o">[]</span> <span class="n">var10000</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">String</span><span class="o">[]{</span><span class="s">"\r\n"</span><span class="o">,</span> <span class="s">"\n"</span><span class="o">,</span> <span class="s">"\r"</span><span class="o">};</span>
<span class="nc">List</span> <span class="n">var1</span><span class="o">;</span>
<span class="nc">DelimitedRangesSequence</span> <span class="n">var2</span><span class="o">;</span>
</code></pre></div></div>
<p>This indicates that we’re using a Kotlin implementation of splitting and using its <code class="highlighter-rouge">Sequence</code> type. Java 11 added a <code class="highlighter-rouge">String.lines()</code> which returns a <code class="highlighter-rouge">Stream</code> that also has the <code class="highlighter-rouge">dropWhile</code> and <code class="highlighter-rouge">takeWhile</code> operators which are already in use. Unfortunately Kotlin also has a <code class="highlighter-rouge">String.lines()</code> extension, so we need a cast in order to use the Java 11 method.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> private fun findDependencyPaths(text: String): Set<List<String>> {
<span class="gd">- val dependencyLines = text.lineSequence()
</span><span class="gi">+ val dependencyLines = (text as java.lang.String).lines()
</span> .dropWhile { !it.startsWith("+--- ") }
.takeWhile { it.isNotEmpty() }
</code></pre></div></div>
<p>This change drops the binary to just 13643 bytes (~13KiB) for a 99.2% reduction.</p>
<h4 id="remaining-bloat">Remaining bloat</h4>
<p>Kotlin being a multiplatform language means that it has its own implementation of an empty list, set, and map. When targeting the JVM, however, there’s no reason to use these over the ones provided by <code class="highlighter-rouge">java.util.Collections</code>. I filed <a href="https://youtrack.jetbrains.com/issue/KT-41333">KT-41333</a> to track this enhancement.</p>
<p>Dumping the contents of the final binary shows its empty collections (and related types) contribute about 50% of the remaining size:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ unzip -l build/libs/dependency-tree-diff-r8.jar
Archive: build/libs/dependency-tree-diff-r8.jar
Length Date Time Name
--------- ---------- ----- ----
84 12-31-1969 19:00 META-INF/MANIFEST.MF
926 12-31-1969 19:00 com/jakewharton/gradle/dependencies/DependencyTrees$findDependencyPaths$dependencyLines$1.class
854 12-31-1969 19:00 com/jakewharton/gradle/dependencies/DependencyTrees$findDependencyPaths$dependencyLines$2.class
6224 12-31-1969 19:00 com/jakewharton/gradle/dependencies/DependencyTreeDiff.class
604 12-31-1969 19:00 com/jakewharton/gradle/dependencies/Node.class
2534 12-31-1969 19:00 kotlin/collections/CollectionsKt__CollectionsKt.class
1120 12-31-1969 19:00 kotlin/collections/EmptyIterator.class
3227 12-31-1969 19:00 kotlin/collections/EmptyList.class
2023 12-31-1969 19:00 kotlin/collections/EmptySet.class
1958 12-31-1969 19:00 kotlin/jvm/internal/CollectionToArray.class
1638 12-31-1969 19:00 kotlin/jvm/internal/Intrinsics.class
--------- -------
21192 11 files
</code></pre></div></div>
<p>In addition to those extra types, the bytecode contains a bunch of extra null checks. For example, the decompiled bytecode for <code class="highlighter-rouge">findDependencyPaths</code> from the last section actually looks like this:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">Set</span> <span class="nf">findDependencyPaths</span><span class="o">(</span><span class="nc">String</span> <span class="n">var0</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">Intrinsics</span><span class="o">.</span><span class="na">checkNotNullParameter</span><span class="o">(</span><span class="n">var0</span><span class="o">,</span> <span class="s">"$this$lines"</span><span class="o">);</span>
<span class="nc">Intrinsics</span><span class="o">.</span><span class="na">checkNotNullParameter</span><span class="o">(</span><span class="n">var0</span><span class="o">,</span> <span class="s">"$this$lineSequence"</span><span class="o">);</span>
<span class="nc">String</span><span class="o">[]</span> <span class="n">var10000</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">String</span><span class="o">[]{</span><span class="s">"\r\n"</span><span class="o">,</span> <span class="s">"\n"</span><span class="o">,</span> <span class="s">"\r"</span><span class="o">};</span>
<span class="nc">Intrinsics</span><span class="o">.</span><span class="na">checkNotNullParameter</span><span class="o">(</span><span class="n">var0</span><span class="o">,</span> <span class="s">"$this$splitToSequence"</span><span class="o">);</span>
<span class="nc">Intrinsics</span><span class="o">.</span><span class="na">checkNotNullParameter</span><span class="o">(</span><span class="n">var10000</span><span class="o">,</span> <span class="s">"delimiters"</span><span class="o">);</span>
<span class="nc">Intrinsics</span><span class="o">.</span><span class="na">checkNotNullParameter</span><span class="o">(</span><span class="n">var10000</span><span class="o">,</span> <span class="s">"$this$asList"</span><span class="o">);</span>
</code></pre></div></div>
<p>These <code class="highlighter-rouge">Intrinsics</code> calls enforce the nullability invariants of the type system on function parameters, but after inlining all but the first one are redundant. Duplicate calls like this appear all over the code. This is <a href="https://issuetracker.google.com/issues/139276374">an R8 bug</a> caused by Kotlin renaming these intrinsic methods and R8 not updating to properly track that change.</p>
<p>With these two issues fixed, it’s likely the binary will drop into single-digit KiBs producing a high-99 percent reduction from the original fat <code class="highlighter-rouge">.jar</code>.</p>
<hr />
<p>If you are building a JVM binary or a JVM library which shades dependencies make sure you use a tool like R8 or ProGuard to remove unused code paths, or use a Graal native image to produce a minimal native binary. This tool was kept as Java bytecode so that a single <code class="highlighter-rouge">.jar</code> can be used on multiple platforms.</p>
<p>The full source code and build setup for <code class="highlighter-rouge">dependency-tree-diff</code> is available <a href="https://github.com/JakeWharton/dependency-tree-diff">on GitHub</a>.</p>
Wire Support For Swift, Part 12020-08-19T00:00:00+00:00https://jakewharton.com/wire-support-for-swift-part-1This post was published externally on Cash App Code Blog. Read it at https://code.cash.app/wire-support-for-swift-part-1.Sixteen corners2020-08-06T00:00:00+00:00https://jakewharton.com/sixteen-corners<p>Last year I built a library called <a href="https://github.com/JakeWharton/picnic">Picnic</a> for rendering data tables in monospaced environments like your terminal. Part of rendering the table is calculating what character to use for each wall and each corner separating the cells.</p>
<p>Here’s a representative output with a bunch of different corner styles:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> │ compressed │ uncompressed
├───────────┬───────────┬───────┼───────────┬───────────┬────────
APK │ old │ new │ diff │ old │ new │ diff
──────────┼───────────┼───────────┼───────┼───────────┼───────────┼────────
dex │ 664.8 KiB │ 664.8 KiB │ -25 B │ 1.5 MiB │ 1.5 MiB │ -112 B
arsc │ 201.7 KiB │ 201.7 KiB │ 0 B │ 201.6 KiB │ 201.6 KiB │ 0 B
manifest │ 1.4 KiB │ 1.4 KiB │ 0 B │ 4.2 KiB │ 4.2 KiB │ 0 B
res │ 418.2 KiB │ 418.2 KiB │ -14 B │ 488.3 KiB │ 488.3 KiB │ 0 B
asset │ 0 B │ 0 B │ 0 B │ 0 B │ 0 B │ 0 B
other │ 37.1 KiB │ 37.1 KiB │ 0 B │ 36.3 KiB │ 36.3 KiB │ 0 B
──────────┼───────────┼───────────┼───────┼───────────┼───────────┼────────
total │ 1.3 MiB │ 1.3 MiB │ -39 B │ 2.2 MiB │ 2.2 MiB │ -112 B
</code></pre></div></div>
<p>Wall border calculation is straightforward. For a vertical wall, a vertical pipe is used if either or both of the two cells wants a border, otherwise an empty space is used.<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup></p>
<p>Corner calculation is a bit more involved. A corner has four potential segments for the four cardinal directions that may be drawn. The four adjacent cells each participate in the visibility of two segments.</p>
<h3 id="corner-characters">Corner Characters</h3>
<p>Once the code determines the four boolean values for the four segments of a corner we need to map that to the display character. Four booleans produce sixteen possible values.</p>
<p>Initially I started with the naive nesting of conditionals to get it working.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">return</span> <span class="k">if</span> <span class="p">(</span><span class="n">left</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">right</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">up</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">down</span><span class="p">)</span> <span class="p">{</span>
<span class="sc">'┼'</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="sc">'┴'</span>
<span class="p">}</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">down</span><span class="p">)</span> <span class="p">{</span>
<span class="sc">'┬'</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span> <span class="cm">/*..*/</span> <span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span> <span class="cm">/*..*/</span> <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Nesting conditionals is an optimization so that each boolean is only checked once. If we wanted, we could flatten the conditionals by repeatedly checking each boolean.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="p">(</span><span class="n">left</span> <span class="p">&&</span> <span class="n">right</span> <span class="p">&&</span> <span class="n">up</span> <span class="p">&&</span> <span class="n">down</span><span class="p">)</span> <span class="k">return</span> <span class="sc">'┼'</span>
<span class="k">if</span> <span class="p">(</span><span class="n">left</span> <span class="p">&&</span> <span class="n">right</span> <span class="p">&&</span> <span class="n">up</span> <span class="p">&&</span> <span class="p">!</span><span class="n">down</span><span class="p">)</span> <span class="k">return</span> <span class="sc">'┴'</span>
<span class="k">if</span> <span class="p">(</span><span class="n">left</span> <span class="p">&&</span> <span class="n">right</span> <span class="p">&&</span> <span class="p">!</span><span class="n">up</span> <span class="p">&&</span> <span class="n">down</span><span class="p">)</span> <span class="k">return</span> <span class="sc">'┬'</span>
<span class="k">if</span> <span class="p">(</span><span class="n">left</span> <span class="p">&&</span> <span class="n">right</span> <span class="p">&&</span> <span class="p">!</span><span class="n">up</span> <span class="p">&&</span> <span class="p">!</span><span class="n">down</span><span class="p">)</span> <span class="k">return</span> <span class="sc">'─'</span>
<span class="c1">// ...</span>
</code></pre></div></div>
<p>The boolean type is a facade over the binary values 0 and 1. Replacing these conditionals with the corresponding binary yields familiar values: <code class="highlighter-rouge">1111</code>, <code class="highlighter-rouge">1110</code>, <code class="highlighter-rouge">1101</code>, <code class="highlighter-rouge">1100</code>, etc. These are the decimal values 15, 14, 13, 12, and so on down to 0.</p>
<p>Mapping the four booleans to these bits gives a decimal we can use to index into a single string which contains all the corner characters.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">val</span> <span class="py">corners</span> <span class="p">=</span> <span class="s">" ╷╵│╶┌└├╴┐┘┤─┬┴┼"</span>
<span class="kd">val</span> <span class="py">index</span> <span class="p">=</span>
<span class="p">(</span><span class="k">if</span> <span class="p">(</span><span class="n">down</span><span class="p">)</span> <span class="mb">0b0001</span> <span class="k">else</span> <span class="mi">0</span><span class="p">)</span> <span class="nf">or</span>
<span class="p">(</span><span class="k">if</span> <span class="p">(</span><span class="n">up</span><span class="p">)</span> <span class="mb">0b0010</span> <span class="k">else</span> <span class="mi">0</span><span class="p">)</span> <span class="nf">or</span>
<span class="p">(</span><span class="k">if</span> <span class="p">(</span><span class="n">right</span><span class="p">)</span> <span class="mb">0b0100</span> <span class="k">else</span> <span class="mi">0</span><span class="p">)</span> <span class="nf">or</span>
<span class="p">(</span><span class="k">if</span> <span class="p">(</span><span class="n">left</span><span class="p">)</span> <span class="mb">0b1000</span> <span class="k">else</span> <span class="mi">0</span><span class="p">)</span>
<span class="k">return</span> <span class="n">corners</span><span class="p">[</span><span class="n">index</span><span class="p">]</span>
</code></pre></div></div>
<p>Much nicer!</p>
<h3 id="testing-corners">Testing Corners</h3>
<p>The logic of determining the four booleans and then choosing the corner character needs tests. Once again I started with the naive approach of a bunch of 2x2 tables with varying borders so that the middle corner was different in each.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@Test</span> <span class="k">fun</span> <span class="nf">borderLeftRightUpDown</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">val</span> <span class="py">table</span> <span class="p">=</span> <span class="nf">table</span> <span class="p">{</span> <span class="cm">/*..*/</span> <span class="p">}</span>
<span class="nf">assertThat</span><span class="p">(</span><span class="n">table</span><span class="p">.</span><span class="nf">renderText</span><span class="p">()).</span><span class="nf">isEqualTo</span><span class="p">(</span><span class="s">"""
|1│2
|─┼─
|3│4
|"""</span><span class="p">.</span><span class="nf">trimMargin</span><span class="p">())</span>
<span class="p">}</span>
<span class="nd">@Test</span> <span class="k">fun</span> <span class="nf">borderLeftRightUp</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">val</span> <span class="py">table</span> <span class="p">=</span> <span class="nf">table</span> <span class="p">{</span> <span class="cm">/*..*/</span> <span class="p">}</span>
<span class="nf">assertThat</span><span class="p">(</span><span class="n">table</span><span class="p">.</span><span class="nf">renderText</span><span class="p">()).</span><span class="nf">isEqualTo</span><span class="p">(</span><span class="s">"""
|1│2
|─┴─
|3 4
|"""</span><span class="p">.</span><span class="nf">trimMargin</span><span class="p">())</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Needing sixteen different tests feels very much like the nested conditionals above. Sure it’s correct, but can we do better? That was the question I presented to two friends who had already been watching me build the library.</p>
<p><a href="/static/post-image/sixteen-corners-test.png"><img src="/static/post-image/sixteen-corners-test.png" alt="Slack message asking whether you can create a 3x3 table that uses all sixteen corner types" width="516" /></a></p>
<p><em>(At this point I think they know to just stand back as I fall down these rabbit holes.)</em></p>
<p>What do you think? Feel free to give it a try! Scroll down for the answer…</p>
<p>1111…</p>
<p>1110…</p>
<p>1101…</p>
<p>1100…</p>
<p>1011…</p>
<p>1010…</p>
<p>1001…</p>
<p>1000…</p>
<p>0111…</p>
<p>0110…</p>
<p>0101…</p>
<p>0100…</p>
<p>0011…</p>
<p>0010…</p>
<p>0001…</p>
<p>0000!</p>
<p>After about 10 minutes at the whiteboard I managed to come up with a configuration that worked.</p>
<p><a href="/static/post-image/sixteen-corners-test-solution.jpg"><img src="/static/post-image/sixteen-corners-test-solution.jpg" alt="Slack message asking whether you can create a 3x3 table that uses all sixteen corner types" width="300" /></a></p>
<p>This translates nicely into a single test.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@Test</span> <span class="k">fun</span> <span class="nf">allCorners</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">val</span> <span class="py">table</span> <span class="p">=</span> <span class="nf">table</span> <span class="p">{</span> <span class="cm">/*..*/</span> <span class="p">}</span>
<span class="nf">assertThat</span><span class="p">(</span><span class="n">table</span><span class="p">.</span><span class="nf">renderText</span><span class="p">()).</span><span class="nf">isEqualTo</span><span class="p">(</span><span class="s">"""
|┌─┬─┐ ╷
|│1│2│3│
|├─┤ ╵ │
|│4│5 6│
|└─┼───┘
| 7│8 9
|╶─┴─╴
|"""</span><span class="p">.</span><span class="nf">trimMargin</span><span class="p">())</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The number of theoretical arrangements of corners is 16!, or 20,922,789,888,000, so finding a solution felt like a nice win.</p>
<p>This post was supposed to stop here, but…</p>
<h3 id="finding-all-possible-arrangements">Finding All Possible Arrangements</h3>
<p>I did the above work a year ago, but upon seeing the <em>very</em> large value of 16! in preparing the post I began to wonder how many valid arrangements exist.</p>
<p>Once again starting naive, I wrote a recursive function which created permutations of the numbers [0,15] and then did a validation pass to see if all corresponding corners had matching edges.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="mi">0</span> <span class="n">until</span> <span class="mi">16</span><span class="p">)</span>
<span class="p">.</span><span class="nf">permutationSequence</span><span class="p">()</span> <span class="c1">// <-- produces Sequence<IntArray></span>
<span class="p">.</span><span class="nf">filter</span> <span class="p">{</span> <span class="nf">validateCorners</span><span class="p">(</span><span class="n">it</span><span class="p">)</span> <span class="p">}</span>
<span class="p">.</span><span class="nf">forEach</span> <span class="p">{</span> <span class="nf">println</span><span class="p">(</span><span class="n">it</span><span class="p">.</span><span class="nf">contentToString</span><span class="p">())</span> <span class="p">}</span>
</code></pre></div></div>
<p>This was exorbitantly slow. I let it run for an hour, and it never got far enough to find a single match.</p>
<p>Instead of validating each complete permutation, huge sets of permutation candidates could immediately be rejected as soon as two corners were invalid. For example, if the very first corner (upper left) has a left or up segment we can immediately reject it and eliminate 15! candidates.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fun</span> <span class="nf">validTables</span><span class="p">():</span> <span class="nc">Sequence</span><span class="p"><</span><span class="nc">IntArray</span><span class="p">></span> <span class="p">=</span> <span class="nf">sequence</span> <span class="p">{</span>
<span class="kd">val</span> <span class="py">state</span> <span class="p">=</span> <span class="nc">IntArray</span><span class="p">(</span><span class="mi">16</span><span class="p">)</span>
<span class="k">suspend</span> <span class="k">fun</span> <span class="nf">SequenceScope</span><span class="p"><</span><span class="nc">IntArray</span><span class="p">>.</span><span class="nf">placeCorner</span><span class="p">(</span><span class="n">index</span><span class="p">:</span> <span class="nc">Int</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">index</span> <span class="p">==</span> <span class="mi">16</span><span class="p">)</span> <span class="p">{</span>
<span class="k">yield</span><span class="p">(</span><span class="n">state</span><span class="p">.</span><span class="nf">clone</span><span class="p">())</span>
<span class="k">return</span>
<span class="p">}</span>
<span class="k">for</span> <span class="p">(</span><span class="n">corner</span> <span class="k">in</span> <span class="mi">0</span> <span class="n">until</span> <span class="mi">16</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// TODO validate corner fits here!</span>
<span class="n">state</span><span class="p">[</span><span class="n">index</span><span class="p">]</span> <span class="p">=</span> <span class="n">corner</span>
<span class="nf">placeCorner</span><span class="p">(</span><span class="n">index</span> <span class="p">+</span> <span class="mi">1</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nf">placeCorner</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Instead of using a two-dimensional array to map the 4x4 grid it is flattened into a single 16-element array.</p>
<p>Since each corner needs to be different, we need to track which of the 16 were already used. This could be done with a <code class="highlighter-rouge">Set<Int></code> but that would require allocation. Since the range of values is [0,15] and we only need to store a boolean value we can once again turn to using bits in a single <code class="highlighter-rouge">Int</code>.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> fun validTables(): Sequence<IntArray> = sequence {
val state = IntArray(16)
<span class="gd">- suspend fun SequenceScope<IntArray>.placeCorner(index: Int) {
</span><span class="gi">+ suspend fun SequenceScope<IntArray>.placeCorner(index: Int, used: Int) {
</span> if (index == 16) {
yield(state.clone())
return
}
for (corner in 0 until 16) {
<span class="gi">+ if (used.hasBit(corner)) continue
+
</span> // TODO validate corner fits here!
state[index] = corner
<span class="gd">- placeCorner(index + 1)
</span><span class="gi">+ placeCorner(index + 1, used.withBit(corner))
</span> }
}
<span class="gd">- placeCorner(0)
</span><span class="gi">+ placeCorner(0, 0)
</span> }
<span class="gi">+
+fun Int.hasBit(bit: Int) = ((1 shl bit) and this) != 0
+fun Int.withBit(bit: Int) = (1 shl bit) or this
</span></code></pre></div></div>
<p>There are three constraints for placing a corner at the current index that must be validated:</p>
<ol>
<li>
<p>If the corner is at the edge of the square, no corner segment must be present in the direction of the edge.</p>
<p>For example, index 0 which is at the top and left edge of the 4x4 cannot be <code class="highlighter-rouge">├</code> because it has an up segment.</p>
</li>
<li>
<p>If there is a corner to the left of the current index in the 4x4 grid, this corner can only have a left segment if that corner has a right segment.</p>
<p>For example, if <code class="highlighter-rouge">┐</code> is at index 1 then <code class="highlighter-rouge">┬</code> is invalid for index 2 since they do not agree about the presence of a horizontal segment.</p>
</li>
<li>
<p>If there is a corner above the current index in the 4x4 grid, this corner can only have an up segment if that corner has a down segment.</p>
<p>For example, if <code class="highlighter-rouge">╶</code> is at index 0 then <code class="highlighter-rouge">├</code> is invalid for index 4 since they do not agree about the presence of a vertical segment.</p>
</li>
</ol>
<p>In the same way four booleans were used as bits to create the numbers [0,15] in the first section, we can invert that operation to extract the four booleans from the numbers to perform validation.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fun</span> <span class="nc">Int</span><span class="p">.</span><span class="nf">hasDownSegment</span><span class="p">()</span> <span class="p">=</span> <span class="p">(</span><span class="mb">0b0001</span> <span class="n">and</span> <span class="k">this</span><span class="p">)</span> <span class="p">!=</span> <span class="mi">0</span>
<span class="k">fun</span> <span class="nc">Int</span><span class="p">.</span><span class="nf">hasUpSegment</span><span class="p">()</span> <span class="p">=</span> <span class="p">(</span><span class="mb">0b0010</span> <span class="n">and</span> <span class="k">this</span><span class="p">)</span> <span class="p">!=</span> <span class="mi">0</span>
<span class="k">fun</span> <span class="nc">Int</span><span class="p">.</span><span class="nf">hasRightSegment</span><span class="p">()</span> <span class="p">=</span> <span class="p">(</span><span class="mb">0b0100</span> <span class="n">and</span> <span class="k">this</span><span class="p">)</span> <span class="p">!=</span> <span class="mi">0</span>
<span class="k">fun</span> <span class="nc">Int</span><span class="p">.</span><span class="nf">hasLeftSegment</span><span class="p">()</span> <span class="p">=</span> <span class="p">(</span><span class="mb">0b1000</span> <span class="n">and</span> <span class="k">this</span><span class="p">)</span> <span class="p">!=</span> <span class="mi">0</span>
</code></pre></div></div>
<p>With these helpers we can add the validation.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> for (corner in 0 until 16) {
if (used.hasBit(corner)) continue
<span class="gd">- // TODO validate corner fits here!
</span><span class="gi">+ if (index > 11 && corner.hasDownSegment()) continue // Bottom row
+ if (index % 4 == 3 && corner.hasRightSegment()) continue // Right column
+
+ // Find the previous row and column corners so we can test if the current corner can fit at
+ // this position. Use 0 when in top row or left column since it will always be incompatible.
+ val previousRowCorner = if (index % 4 == 0) 0 else state[index - 1]
+ val previousColCorner = if (index < 4) 0 else state[index - 4]
+
+ if (previousRowCorner.hasRightSegment() != corner.hasLeftSegment()) continue // Horizontal mismatch
+ if (previousColCorner.hasDownSegment() != corner.hasUpSegment()) continue // Vertical mismatch
</span>
state[index] = i
placeCorner(index + 1, used.withBit(i))
}
</code></pre></div></div>
<p>With no allocation and being able to quickly reject massive sets of invalid candidates this should hopefully produce results in less than an hour. Let’s run it!</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fun</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">val</span> <span class="py">time</span> <span class="p">=</span> <span class="nf">measureTimeMillis</span> <span class="p">{</span>
<span class="nf">validTables</span><span class="p">().</span><span class="nf">forEachIndexed</span> <span class="p">{</span> <span class="n">index</span><span class="p">,</span> <span class="n">corners</span> <span class="p">-></span>
<span class="kd">val</span> <span class="py">table</span> <span class="p">=</span> <span class="n">corners</span><span class="p">.</span><span class="nf">map</span> <span class="p">{</span> <span class="s">" ╷╵│╶┌└├╴┐┘┤─┬┴┼"</span><span class="p">.</span><span class="k">get</span><span class="p">(</span><span class="n">it</span><span class="p">)</span> <span class="p">}</span>
<span class="p">.</span><span class="nf">joinToString</span><span class="p">(</span><span class="s">""</span><span class="p">)</span>
<span class="p">.</span><span class="nf">chunked</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span>
<span class="p">.</span><span class="nf">joinToString</span><span class="p">(</span><span class="s">"\n"</span><span class="p">)</span>
<span class="nf">println</span><span class="p">(</span><span class="s">"#${index + 1}: ${state.contentToString()}\n$table\n"</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nf">println</span><span class="p">(</span><span class="s">"Done. Took $time milliseconds."</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Survey says?</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#1: [0, 1, 4, 8, 5, 15, 12, 9, 3, 7, 13, 11, 2, 6, 14, 10]
╷╶╴
┌┼─┐
│├┬┤
╵└┴┘
...
#652: [5, 13, 12, 9, 7, 15, 8, 3, 6, 11, 1, 2, 4, 14, 10, 0]
┌┬─┐
├┼╴│
└┤╷╵
╶┴┘
Done. Took 57 milliseconds.
</code></pre></div></div>
<p>Considerably faster than taking hours! Only 652 valid candidates out of 20,922,789,888,000 possible permutations. You can check out the full list <a href="https://gist.github.com/JakeWharton/e7bea82598695adeab244f194be07228#file-valid-txt">here</a>.</p>
<p>If we look at output #1 above, this table is <em>technically</em> invalid since it contains an orphan corner in the upper right. There is no way to create such a corner by setting borders on table cells. However, purely from a segment-validation standpoint it is valid. Visual inspection of the candidates makes it seem like about 15-25% suffer from this case.</p>
<p>I’m out of time on this post, so finding the true number of valid configurations expressible by table cell borders will have to be an exercise left to the reader.</p>
<hr />
<p>Creating <a href="https://github.com/JakeWharton/picnic">Picnic</a> was a fun rabbit hole to fall into for a few days last year. Aside from the challenges of corners, it implements the CSS specification for measuring and laying out tables and supports row and column spans, vertical and horizontal text alignment, and vertical and horizontal cell padding.</p>
<p>If you ever need to display a command-line table and have written an HTML table in your life it should be very approachable with Picnic.</p>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>It’s actually little more complicated than this. If none of the rows want to draw a border between the two cells in these columns then the border width will be zero and won’t occupy any space. <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
R8 Optimization: Lambda Groups2020-04-30T00:00:00+00:00https://jakewharton.com/r8-optimization-lambda-groups<blockquote>
<p>Note: This post is part of a series on D8 and R8, Android’s new dexer and optimizer, respectively. For an intro to D8 read <a href="/androids-java-8-support/">“Android’s Java 8 support”</a>. For an intro to R8 read <a href="/r8-optimization-staticization/">“R8 Optimization: Staticization”</a>.</p>
</blockquote>
<p>Lambda usage in Kotlin feels more pervasive than Java because of the functional nature of the Kotlin standard library. Some lambdas are merely syntactic constructs that are eliminated at compile-time through the use of <code class="highlighter-rouge">inline</code> functions. The rest materialize into whole classes for use at runtime.</p>
<p>The mechanisms by which lambdas work was covered in the <a href="/androids-java-8-support/">Android Java 8 support</a> post, but here’s a quick refresher:</p>
<ul>
<li><code class="highlighter-rouge">javac</code> hoists lambda bodies to a package-private method and writes an <code class="highlighter-rouge">invoke-dynamic</code> bytecode for the target lambda type at the call-site. The JVM spins a class at runtime of the desired type and invokes the package-private method in the method body. Android does not ship this runtime support, so D8 performs a compile-time transformation to a class which implements the desired type and which invokes the package-private method.</li>
<li><code class="highlighter-rouge">kotlinc</code> skips the <code class="highlighter-rouge">invoke-dynamic</code> bytecode (even when targeting Java 8+) and generates full classes directly.</li>
</ul>
<p>Here’s two Kotlin classes and some lambda usage that we can experiment with.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Employee</span><span class="p">(</span>
<span class="kd">val</span> <span class="py">id</span><span class="p">:</span> <span class="nc">String</span><span class="p">,</span>
<span class="kd">val</span> <span class="py">joined</span><span class="p">:</span> <span class="nc">LocalDate</span><span class="p">,</span>
<span class="kd">val</span> <span class="py">managerId</span><span class="p">:</span> <span class="nc">String</span><span class="p">?</span>
<span class="p">)</span>
<span class="kd">class</span> <span class="nc">EmployeeRepository</span><span class="p">(</span><span class="kd">val</span> <span class="py">allEmployees</span><span class="p">:</span> <span class="p">()</span> <span class="p">-></span> <span class="nc">Sequence</span><span class="p"><</span><span class="nc">Employee</span><span class="p">>)</span> <span class="p">{</span>
<span class="k">fun</span> <span class="nf">joinedAfter</span><span class="p">(</span><span class="n">date</span><span class="p">:</span> <span class="nc">LocalDate</span><span class="p">)</span> <span class="p">=</span>
<span class="nf">allEmployees</span><span class="p">()</span>
<span class="p">.</span><span class="nf">filter</span> <span class="p">{</span> <span class="n">it</span><span class="p">.</span><span class="n">joined</span> <span class="p">>=</span> <span class="n">date</span> <span class="p">}</span>
<span class="p">.</span><span class="nf">toList</span><span class="p">()</span>
<span class="k">fun</span> <span class="nf">reports</span><span class="p">(</span><span class="n">manager</span><span class="p">:</span> <span class="nc">Employee</span><span class="p">)</span> <span class="p">=</span>
<span class="nf">allEmployees</span><span class="p">()</span>
<span class="p">.</span><span class="nf">filter</span> <span class="p">{</span> <span class="n">it</span><span class="p">.</span><span class="n">managerId</span> <span class="p">==</span> <span class="n">manager</span><span class="p">.</span><span class="n">id</span> <span class="p">}</span>
<span class="p">.</span><span class="nf">toList</span><span class="p">()</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The <code class="highlighter-rouge">EmployeeRepository</code> class accepts a lambda which produces a sequence of employees and exposes two functions for listing the employees who joined after a particular date and those who report to a particular employee. Both functions use a lambda to filter the sequence to the desired items before converting to a list.</p>
<p>Kotlin’s approach to lambdas is immediately visible after compiling this class.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ kotlinc EmployeeRepository.kt
$ ls *.class
Employee.class
EmployeeRepository.class
EmployeeRepository$joinedAfter$1.class
EmployeeRepository$reports$1.class
</code></pre></div></div>
<p>Each lambda has a unique name formed by joining the enclosing class name, enclosing function name, and a monotonic value.</p>
<h3 id="kotlin-lambdas-and-d8">Kotlin Lambdas and D8</h3>
<p>To establish a baseline of what ends up in our APK, let’s run these classfiles through D8.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ java -jar $R8_HOME/build/libs/d8.jar \
--lib $ANDROID_HOME/platforms/android-29/android.jar \
--release \
--output . \
*.class
</code></pre></div></div>
<p>You can dump the whole output with <code class="highlighter-rouge">dexdump -d classes.dex</code>, but let’s focus on the bodies of the <code class="highlighter-rouge">joinedAfter</code> and <code class="highlighter-rouge">reports</code> functions.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[000590] EmployeeRepository.joinedAfter:(Ljava/time/LocalDate;)Ljava/util/List;
0000: iget-object v0, v2, LEmployeeRepository;.allEmployees:Lkotlin/jvm/functions/Function0;
0002: invoke-interface {v0}, Lkotlin/jvm/functions/Function0;.invoke:()Ljava/lang/Object;
0005: move-result-object v0
0006: new-instance v1, LEmployeeRepository$joinedAfter$1;
0008: invoke-direct {v1, v3}, LEmployeeRepository$joinedAfter$1;.<init>:(Ljava/time/LocalDate;)V
000b: invoke-static {v0, v1}, Lkotlin/sequences/SequencesKt;.filter:(Lkotlin/sequences/Sequence;Lkotlin/jvm/functions/Function1;)Lkotlin/sequences/Sequence;
000e: move-result-object v0
000f: invoke-static {v0}, Lkotlin/sequences/SequencesKt;.toList:(Lkotlin/sequences/Sequence;)Ljava/util/List;
0012: move-result-object v0
0013: return-object v0
[0005dc] EmployeeRepository.reports:(LEmployee;)Ljava/util/List;
0000: iget-object v0, v2, LEmployeeRepository;.allEmployees:Lkotlin/jvm/functions/Function0;
0002: invoke-interface {v0}, Lkotlin/jvm/functions/Function0;.invoke:()Ljava/lang/Object;
0005: move-result-object v0
0006: new-instance v1, LEmployeeRepository$reports$1;
0008: invoke-direct {v1, v3}, LEmployeeRepository$reports$1;.<init>:(LEmployee;)V
000b: invoke-static {v0, v1}, Lkotlin/sequences/SequencesKt;.filter:(Lkotlin/sequences/Sequence;Lkotlin/jvm/functions/Function1;)Lkotlin/sequences/Sequence;
000e: move-result-object v0
000f: invoke-static {v0}, Lkotlin/sequences/SequencesKt;.toList:(Lkotlin/sequences/Sequence;)Ljava/util/List;
0012: move-result-object v0
0013: return-object v0
</code></pre></div></div>
<p>There’s a lot going on here, but each function is almost identical so we can break both down at once:</p>
<ul>
<li><code class="highlighter-rouge">0000</code>-<code class="highlighter-rouge">0005</code> gets the <code class="highlighter-rouge">Sequence<Employee></code> by invoking the <code class="highlighter-rouge">allEmployees</code> lambda.</li>
<li><code class="highlighter-rouge">0006</code> creates an instance of the respective lambda class for each function.</li>
<li><code class="highlighter-rouge">0008</code> calls the lambda class constructor, passing in either the date or manager argument as the sole parameter.</li>
<li><code class="highlighter-rouge">000b</code>-<code class="highlighter-rouge">000e</code> calls <code class="highlighter-rouge">filter</code> on the sequence passing in the lambda instance.</li>
<li><code class="highlighter-rouge">000f</code>-<code class="highlighter-rouge">0012</code> calls <code class="highlighter-rouge">toList</code> on the filtered sequence.</li>
<li><code class="highlighter-rouge">0013</code> returns the list.</li>
</ul>
<p>If we looked at the lambda classes we would find each implementing the <code class="highlighter-rouge">Function1</code> interface, having a field of type <code class="highlighter-rouge">LocalDate</code> or <code class="highlighter-rouge">Employee</code>, having a constructor which accepts a parameter and sets its field, and having an <code class="highlighter-rouge">invoke</code> method with the body of the lambda.</p>
<p>D8 performs a straightforward translation of the Java bytecode into the equivalent Dalvik bytecode. It’s only when we break out R8 do interesting things start to happen.</p>
<h3 id="kotlin-lambdas-and-r8">Kotlin Lambdas and R8</h3>
<p>Since we have no actual usage of these APIs, they need explicitly kept or R8 will produce an empty dex file.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-keep class Employee { *; }
-keep class EmployeeRepository { *; }
-dontobfuscate
</code></pre></div></div>
<p>With our two classes kept, let’s run R8 and see what changes.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ java -jar $R8_HOME/build/libs/r8.jar \
--lib $ANDROID_HOME/platforms/android-29/android.jar \
--release \
--output . \
--pg-conf rules.txt \
*.class kotlin-stdlib-*.jar
</code></pre></div></div>
<p>We can see what has changed in the bodies of the <code class="highlighter-rouge">joinedAfter</code> and <code class="highlighter-rouge">reports</code> functions.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> [000dd4] EmployeeRepository.joinedAfter:(Ljava/time/LocalDate;)Ljava/util/List;
0000: iget-object v0, v3, LEmployeeRepository;.allEmployees:Lkotlin/jvm/functions/Function0;
0002: invoke-interface {v0}, Lkotlin/jvm/functions/Function0;.invoke:()Ljava/lang/Object;
0005: move-result-object v0
<span class="gd">-0006: new-instance v1, LEmployeeRepository$joinedAfter$1;
-0008: invoke-direct {v1, v3}, LEmployeeRepository$joinedAfter$1;.<init>:(Ljava/time/LocalDate;)V
</span><span class="gi">+0006: new-instance v1, L-$$LambdaGroup$ks$D2r6uJKXMyXfodlTO7Kw1WcCloA;
+0008: const/4 v2, #int 0
+0009: invoke-direct {v1, v2, v4}, L-$$LambdaGroup$ks$D2r6uJKXMyXfodlTO7Kw1WcCloA;.<init>:(ILjava/lang/Object;)V
</span> 000d: invoke-static {v0, v1}, Lkotlin/sequences/SequencesKt;.filter:(Lkotlin/sequences/Sequence;Lkotlin/jvm/functions/Function1;)Lkotlin/sequences/Sequence;
0010: move-result-object v0
0011: invoke-static {v0}, Lkotlin/sequences/SequencesKt;.toList:(Lkotlin/sequences/Sequence;)Ljava/util/List;
0014: move-result-object v0
0015: return-object v0
[000e34] EmployeeRepository.reports:(LEmployee;)Ljava/util/List;
0000: iget-object v0, v3, LEmployeeRepository;.allEmployees:Lkotlin/jvm/functions/Function0;
0002: invoke-interface {v0}, Lkotlin/jvm/functions/Function0;.invoke:()Ljava/lang/Object;
0005: move-result-object v0
<span class="gd">-0006: new-instance v1, LEmployeeRepository$reports$1;
-0008: invoke-direct {v1, v3}, LEmployeeRepository$reports$1;.<init>:(LEmployee;)V
</span><span class="gi">+0006: new-instance v1, L-$$LambdaGroup$ks$D2r6uJKXMyXfodlTO7Kw1WcCloA;
+0008: const/4 v2, #int 1
+0009: invoke-direct {v1, v2, v4}, L-$$LambdaGroup$ks$D2r6uJKXMyXfodlTO7Kw1WcCloA;.<init>:(ILjava/lang/Object;)V
</span> 000d: invoke-static {v0, v1}, Lkotlin/sequences/SequencesKt;.filter:(Lkotlin/sequences/Sequence;Lkotlin/jvm/functions/Function1;)Lkotlin/sequences/Sequence;
0010: move-result-object v0
0011: invoke-static {v0}, Lkotlin/sequences/SequencesKt;.toList:(Lkotlin/sequences/Sequence;)Ljava/util/List;
0014: move-result-object v0
0015: return-object v0
</code></pre></div></div>
<p>Let’s break down the new bytecode:</p>
<ul>
<li><code class="highlighter-rouge">0006</code> creates an instance of a class named <code class="highlighter-rouge">-$$LambdaGroup$ks$D2r6uJKXMyXfodlTO7Kw1WcCloA</code>. And, notably, both functions are creating an instance of the <strong>same class</strong> now.</li>
<li><code class="highlighter-rouge">0008</code> stores an integer value of 0 for <code class="highlighter-rouge">joinedAfter</code> and 1 for <code class="highlighter-rouge">reports</code>.</li>
<li><code class="highlighter-rouge">0009</code> call the class constructor and passes the integer and either the date or manager (but as an <code class="highlighter-rouge">Object</code>).</li>
</ul>
<p>Both functions are now instantiating the same class for their lambda. Let’s peek at that class.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Class #15 -
Class descriptor : 'L-$$LambdaGroup$ks$D2r6uJKXMyXfodlTO7Kw1WcCloA;'
Access flags : 0x0011 (PUBLIC FINAL)
Interfaces -
#0 : 'Lkotlin/jvm/functions/Function1;'
Instance fields -
#0 : (in L-$$LambdaGroup$ks$D2r6uJKXMyXfodlTO7Kw1WcCloA;)
name : '$capture$0'
type : 'Ljava/lang/Object;'
access : 0x1011 (PUBLIC FINAL SYNTHETIC)
#1 : (in L-$$LambdaGroup$ks$D2r6uJKXMyXfodlTO7Kw1WcCloA;)
name : '$id$'
type : 'I'
access : 0x1011 (PUBLIC FINAL SYNTHETIC)
Direct methods -
#0 : (in L-$$LambdaGroup$ks$D2r6uJKXMyXfodlTO7Kw1WcCloA;)
name : '<init>'
type : '(ILjava/lang/Object;)V'
access : 0x10001 (PUBLIC CONSTRUCTOR)
code -
[000db0] -$$LambdaGroup$ks$D2r6uJKXMyXfodlTO7Kw1WcCloA.<init>:(ILjava/lang/Object;)V
0000: iput v1, v0, L-$$LambdaGroup$ks$D2r6uJKXMyXfodlTO7Kw1WcCloA;.$id$:I
0002: iput-object v2, v0, L-$$LambdaGroup$ks$D2r6uJKXMyXfodlTO7Kw1WcCloA;.$capture$0:Ljava/lang/Object;
0004: return-void
</code></pre></div></div>
<p>This output tells us that the class implements the <code class="highlighter-rouge">Function1</code> interface, has two fields: an object and integer <code class="highlighter-rouge">id</code>, and has a constructor which accepts an object and integer and assigns the two fields.</p>
<p>Now let’s look at the implementation of its<code class="highlighter-rouge">invoke</code> function.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[000d14] -$$LambdaGroup$ks$D2r6uJKXMyXfodlTO7Kw1WcCloA.invoke:(Ljava/lang/Object;)Ljava/lang/Object;
0000: iget v0, v4, L-$$LambdaGroup$ks$D2r6uJKXMyXfodlTO7Kw1WcCloA;.$id$:I
0002: iget-object v1, v4, L-$$LambdaGroup$ks$D2r6uJKXMyXfodlTO7Kw1WcCloA;.$capture$0:Ljava/lang/Object;
0004: if-eqz v0, 002c
0006: const/4 v2, #int 1
0007: if-ne v0, v2, 002a
000a: check-cast v1, LEmployee;
⋮
0029: return-object v5
002a: const/4 v5, #int 0
002b: throw v5
002c: check-cast v0, Ljava/time/LocalDate;
⋮
0044: return-object v5
</code></pre></div></div>
<p>I’ve trimmed quite a lot, but let’s break it down:</p>
<ul>
<li><code class="highlighter-rouge">0000</code> loads the integer <code class="highlighter-rouge">id</code> value from the field.</li>
<li><code class="highlighter-rouge">0002</code> loads the object value from the field.</li>
<li><code class="highlighter-rouge">0004</code> checks if the <code class="highlighter-rouge">id</code> is zero and if so jumps to <code class="highlighter-rouge">002c</code>.</li>
<li><code class="highlighter-rouge">0006</code>-<code class="highlighter-rouge">0007</code> checks if the <code class="highlighter-rouge">id</code> is <em>not</em> one and if so jumps to <code class="highlighter-rouge">002a</code>.</li>
<li><code class="highlighter-rouge">000a</code>-<code class="highlighter-rouge">0029</code> casts the object to <code class="highlighter-rouge">Employee</code> and runs the code from the <code class="highlighter-rouge">reports</code> lambda body. Remember, this codepath is taken if the previous comparison of <code class="highlighter-rouge">id != 1</code> <em>fails</em>.</li>
<li><code class="highlighter-rouge">002a</code>-<code class="highlighter-rouge">002a</code> causes a <code class="highlighter-rouge">NullPointerException</code>. Remember, this codepath is taken if <code class="highlighter-rouge">id</code> is not zero and not one.</li>
<li><code class="highlighter-rouge">002c</code>-<code class="highlighter-rouge">0044</code> casts the object to <code class="highlighter-rouge">LocalDate</code> and runs the code from the <code class="highlighter-rouge">joinedAfter</code> lambda body. Remember, this codepath is taken if <code class="highlighter-rouge">id</code> is zero.</li>
</ul>
<p>It can be hard to follow exactly what this transformation means solely by looking at Dalvik bytecode. We can make the equivalent transformation in source code to illustrate it more clearly.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> class EmployeeRepository(val allEmployees: () -> Sequence<Employee>) {
fun joinedAfter(date: LocalDate) =
allEmployees()
<span class="gd">- .filter { it.joined >= date }
</span><span class="gi">+ .fitler(MyLambdaGroup(date, 0))
</span> .toList()
fun reports(manager: Employee) =
allEmployees()
<span class="gd">- .filter { it.managerId == manager.id }
</span><span class="gi">+ .filter(MyLambdaGroup(manager, 1))
</span> .toList()
}
<span class="gi">+
+private class MyLambdaGroup(
+ private val capture0: Any?,
+ private val id: Int
+) : (Employee) -> Boolean {
+ override fun invoke(employee: Employee): Boolean
+ return when (id) {
+ 0 -> employee.joinedAfter >= (capture0 as LocalDate)
+ 1 -> employee.managerId == (capture0 as Employee).id
+ else -> throw NullPointerException()
+ }
+ }
+}
</span></code></pre></div></div>
<p>The two lambdas which would have produced two classes have been replaced by a single class with an integer discriminator for its behavior. By merging the bodies of the lambdas, the number of classes in the APK can be reduced.</p>
<p>This only works because the two lambdas have the same shape. They do not need to be <em>exactly</em> the same as we can see in our example. One lambda captures a <code class="highlighter-rouge">LocalDate</code> but the other captures an <code class="highlighter-rouge">Employee</code>. Since both only capture a single value they have the same shape and can be merged into this single “lambda group” class.</p>
<h3 id="java-lambdas-and-r8">Java Lambdas and R8</h3>
<p>Let’s rewrite our repository in Java and see what happens.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">final</span> <span class="kd">class</span> <span class="nc">EmployeeRepository</span> <span class="o">{</span>
<span class="kd">private</span> <span class="kd">final</span> <span class="nc">Function0</span><span class="o"><</span><span class="nc">Sequence</span><span class="o"><</span><span class="nc">Employee</span><span class="o">>></span><span class="n">allEmployees</span><span class="o">;</span>
<span class="nc">EmployeeRepository</span><span class="o">(</span><span class="nc">Function0</span><span class="o"><</span><span class="nc">Sequence</span><span class="o"><</span><span class="nc">Employee</span><span class="o">>></span> <span class="n">allEmployees</span><span class="o">)</span> <span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">allEmployees</span> <span class="o">=</span> <span class="n">allEmployees</span><span class="o">;</span>
<span class="o">}</span>
<span class="nc">List</span><span class="o"><</span><span class="nc">Employee</span><span class="o">></span> <span class="nf">joinedAfter</span><span class="o">(</span><span class="nc">LocalDate</span> <span class="n">date</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="nc">SequencesKt</span><span class="o">.</span><span class="na">toList</span><span class="o">(</span>
<span class="nc">SequencesKt</span><span class="o">.</span><span class="na">filter</span><span class="o">(</span>
<span class="n">allEmployees</span><span class="o">.</span><span class="na">invoke</span><span class="o">(),</span>
<span class="n">e</span> <span class="o">-></span> <span class="n">e</span><span class="o">.</span><span class="na">getJoined</span><span class="o">().</span><span class="na">compareTo</span><span class="o">(</span><span class="n">date</span><span class="o">)</span> <span class="o">>=</span> <span class="mi">0</span><span class="o">));</span>
<span class="o">}</span>
<span class="nc">List</span><span class="o"><</span><span class="nc">Employee</span><span class="o">></span> <span class="nf">reports</span><span class="o">(</span><span class="nc">Employee</span> <span class="n">manager</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="nc">SequencesKt</span><span class="o">.</span><span class="na">toList</span><span class="o">(</span>
<span class="nc">SequencesKt</span><span class="o">.</span><span class="na">filter</span><span class="o">(</span>
<span class="n">allEmployees</span><span class="o">.</span><span class="na">invoke</span><span class="o">(),</span>
<span class="n">e</span> <span class="o">-></span> <span class="nc">Objects</span><span class="o">.</span><span class="na">equals</span><span class="o">(</span><span class="n">e</span><span class="o">.</span><span class="na">getManagerId</span><span class="o">(),</span> <span class="n">manager</span><span class="o">.</span><span class="na">getId</span><span class="o">())));</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>We’re using Kotlin’s <code class="highlighter-rouge">Function0</code> instead of <code class="highlighter-rouge">Supplier</code>, <code class="highlighter-rouge">Sequence</code> instead of <code class="highlighter-rouge">Stream</code>, and sequence extensions as static helpers to keep the two examples as close to each other as possible. We can compile with <code class="highlighter-rouge">javac</code> and reuse the same R8 invocation.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ rm EmployeeRepository*.class
$ javac -cp . EmployeeRepository.class
$ java -jar $R8_HOME/build/libs/r8.jar \
--lib $ANDROID_HOME/platforms/android-29/android.jar \
--release \
--output . \
--pg-conf rules.txt \
*.class kotlin-stdlib-*.jar
</code></pre></div></div>
<p>The <code class="highlighter-rouge">joinedAfter</code> and <code class="highlighter-rouge">reports</code> function bodies should look the same as when they were written in Kotlin, right?</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[000d2c] EmployeeRepository.joinedAfter:(Ljava/time/LocalDate;)Ljava/util/List;
⋮
0008: new-instance v1, L-$$Lambda$EmployeeRepository$RwNrgP_DBeZWqltgaXgoLCrPfqI;
000a: invoke-direct {v1, v4}, L-$$Lambda$EmployeeRepository$RwNrgP_DBeZWqltgaXgoLCrPfqI;.<init>:(Ljava/time/LocalDate;)V
⋮
[000d80] EmployeeRepository.reports:(LEmployee;)Ljava/util/List;
⋮
0008: new-instance v1, L-$$Lambda$EmployeeRepository$JjZ4a6TbrR3768PIUyNflFlLVF8;
000a: invoke-direct {v1, v4}, L-$$Lambda$EmployeeRepository$JjZ4a6TbrR3768PIUyNflFlLVF8;.<init>:(LEmployee;)V
⋮
</code></pre></div></div>
<p>They do not! Each implementation is calling into its own lambda class rather than using a lambda group.</p>
<p>As far as I can tell, there’s no technical limitation as to why this would only work for Kotlin lambdas but not Java lambdas. The work just hasn’t been done yet. <a href="https://issuetracker.google.com/issues/153773246">Issue 153773246</a> tracks adding support for merging Java lambdas into lambda groups.</p>
<hr />
<p>By merging lambdas of the same shape together, R8 reduces the APK size impact and runtime classloading burden at the expense of increasing the method body of the lambda.</p>
<p>While the optimization does run on the entire app, by default merging will only occur within a package. This ensures any package-private methods or types used in the lambda body are accessible. Add the <code class="highlighter-rouge">-allowaccessmodification</code> directive to your shrinker rules to enable R8 to globally merge lambdas by increasing the visibility of referenced methods and types when needed.</p>
<p>You may have noticed that the names of the classes generated for Java lambdas and lambda groups appear to have some kind of hash in them. In the next post we’re going to dig into the unique naming of these classes.</p>
Which is better on Android: divide by 2 or shift by 1?2020-04-23T00:00:00+00:00https://jakewharton.com/which-is-better-on-android-divide-by-two-or-shift-by-one<p>I’ve been porting the <a href="https://developer.android.com/reference/androidx/collection/package-summary">AndroidX collection library</a> to Kotlin multiplatform to experiment with binary compatibility, performance, tooling, and the different memory models. Some of the data structures in the library use array-based binary trees to store elements. The Java code has a lot of shifts to replace power-of-two multiplications and divides. When ported to Kotlin, these turn into the slightly-awkward infix operators which further obfuscate the intent of the code.</p>
<p>I sampled a few people about bitwise shifts vs. multiplication/division and many had heard anecdotal claims of shifts having better performance, but everyone remained skeptical of whether it was true. Some assumed that one of the compilers seen before the code ran on a CPU would handle optimizing this case.</p>
<p>In an effort to satisfy my curiosity (and partially to avoid Kotlin’s infix bitwise operators) I set out to answer which is better and some other related questions. Let’s go!</p>
<h3 id="does-anyone-optimize-this">Does anyone optimize this?</h3>
<p>There are three major compilers that code passes through before it hits the CPU: <code class="highlighter-rouge">javac</code>/<code class="highlighter-rouge">kotlinc</code>, D8/R8, and ART.</p>
<p><a href="/static/post-image/android_compilation_pipeline.png"><img src="/static/post-image/android_compilation_pipeline.png" width="788" /></a></p>
<p>Each of these has the opportunity to optimize. But do they?</p>
<h4 id="javac">javac</h4>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Example</span> <span class="o">{</span>
<span class="kd">static</span> <span class="kt">int</span> <span class="nf">multiply</span><span class="o">(</span><span class="kt">int</span> <span class="n">value</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">value</span> <span class="o">*</span> <span class="mi">2</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">static</span> <span class="kt">int</span> <span class="nf">divide</span><span class="o">(</span><span class="kt">int</span> <span class="n">value</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">value</span> <span class="o">/</span> <span class="mi">2</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">static</span> <span class="kt">int</span> <span class="nf">shiftLeft</span><span class="o">(</span><span class="kt">int</span> <span class="n">value</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">value</span> <span class="o"><<</span> <span class="mi">1</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">static</span> <span class="kt">int</span> <span class="nf">shiftRight</span><span class="o">(</span><span class="kt">int</span> <span class="n">value</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">value</span> <span class="o">>></span> <span class="mi">1</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>This Java can be compiled with <code class="highlighter-rouge">javac</code> from JDK 14 and the resulting bytecode can be displayed with <code class="highlighter-rouge">javap</code>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ javac Example.java
$ javap -c Example
Compiled from "Example.java"
class Example {
static int multiply(int);
Code:
0: iload_0
1: iconst_2
2: imul
3: ireturn
static int divide(int);
Code:
0: iload_0
1: iconst_2
2: idiv
3: ireturn
static int shiftLeft(int);
Code:
0: iload_0
1: iconst_1
2: ishl
3: ireturn
static int shiftRight(int);
Code:
0: iload_0
1: iconst_1
2: ishr
3: ireturn
}
</code></pre></div></div>
<p>Every method starts with <code class="highlighter-rouge">iload_0</code> which loads the first argument value. The multiply and divide methods both then have <code class="highlighter-rouge">iconst_2</code> which loads the constant value 2. Each then runs <code class="highlighter-rouge">imul</code> or <code class="highlighter-rouge">idiv</code> to perform integer multiplication or integer division, respectively. The shift methods load the constant value 1 before <code class="highlighter-rouge">ishl</code> or <code class="highlighter-rouge">ishr</code> which is an integer shift left or integer shift right, respectively.</p>
<p>No optimization here, but if you know anything about Java this isn’t unexpected. <code class="highlighter-rouge">javac</code> isn’t an optimizing compiler and leaves the majority of the work to its runtime compilers on the JVM or ahead-of-time compilers.</p>
<h4 id="kotlinc">kotlinc</h4>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fun</span> <span class="nf">multiply</span><span class="p">(</span><span class="n">value</span><span class="p">:</span> <span class="nc">Int</span><span class="p">)</span> <span class="p">=</span> <span class="n">value</span> <span class="p">*</span> <span class="mi">2</span>
<span class="k">fun</span> <span class="nf">divide</span><span class="p">(</span><span class="n">value</span><span class="p">:</span> <span class="nc">Int</span><span class="p">)</span> <span class="p">=</span> <span class="n">value</span> <span class="p">/</span> <span class="mi">2</span>
<span class="k">fun</span> <span class="nf">shiftLeft</span><span class="p">(</span><span class="n">value</span><span class="p">:</span> <span class="nc">Int</span><span class="p">)</span> <span class="p">=</span> <span class="n">value</span> <span class="n">shl</span> <span class="mi">1</span>
<span class="k">fun</span> <span class="nf">shiftRight</span><span class="p">(</span><span class="n">value</span><span class="p">:</span> <span class="nc">Int</span><span class="p">)</span> <span class="p">=</span> <span class="n">value</span> <span class="n">shr</span> <span class="mi">1</span>
</code></pre></div></div>
<p>The Kotlin is compiled to Java bytecode with <code class="highlighter-rouge">kotlinc</code> from Kotlin 1.4-M1 where the <code class="highlighter-rouge">javap</code> tool can once again be used.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ kotlinc Example.kt
$ javap -c ExampleKt
Compiled from "Example.kt"
public final class ExampleKt {
public static final int multiply(int);
Code:
0: iload_0
1: iconst_2
2: imul
3: ireturn
public static final int divide(int);
Code:
0: iload_0
1: iconst_2
2: idiv
3: ireturn
public static final int shiftLeft(int);
Code:
0: iload_0
1: iconst_1
2: ishl
3: ireturn
public static final int shiftRight(int);
Code:
0: iload_0
1: iconst_1
2: ishr
3: ireturn
}
</code></pre></div></div>
<p>Exactly the same output as Java. This is using the original JVM backend of Kotlin, but using the forthcoming IR-based backend (via <code class="highlighter-rouge">-Xuse-ir</code>) also produces the same output.</p>
<h4 id="d8">D8</h4>
<p>We’ll use the Java bytecode output from the Kotlin example as input to the latest D8 built from <code class="highlighter-rouge">master</code> (SHA <code class="highlighter-rouge">2a2bf622d</code> at the time of writing).</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ java -jar $R8_HOME/build/libs/d8.jar \
--release \
--output . \
ExampleKt.class
$ dexdump -d classes.dex
Opened 'classes.dex', DEX version '035'
Class #0 -
Class descriptor : 'LExampleKt;'
Access flags : 0x0011 (PUBLIC FINAL)
Superclass : 'Ljava/lang/Object;'
Direct methods -
#0 : (in LExampleKt;)
name : 'divide'
type : '(I)I'
access : 0x0019 (PUBLIC STATIC FINAL)
code -
000118: |[000118] ExampleKt.divide:(I)I
000128: db00 0102 |0000: div-int/lit8 v0, v1, #int 2 // #02
00012c: 0f00 |0002: return v0
#1 : (in LExampleKt;)
name : 'multiply'
type : '(I)I'
access : 0x0019 (PUBLIC STATIC FINAL)
code -
000130: |[000130] ExampleKt.multiply:(I)I
000140: da00 0102 |0000: mul-int/lit8 v0, v1, #int 2 // #02
000144: 0f00 |0002: return v0
#2 : (in LExampleKt;)
name : 'shiftLeft'
type : '(I)I'
access : 0x0019 (PUBLIC STATIC FINAL)
code -
000148: |[000148] ExampleKt.shiftLeft:(I)I
000158: e000 0101 |0000: shl-int/lit8 v0, v1, #int 1 // #01
00015c: 0f00 |0002: return v0
#3 : (in LExampleKt;)
name : 'shiftRight'
type : '(I)I'
access : 0x0019 (PUBLIC STATIC FINAL)
code -
000160: |[000160] ExampleKt.shiftRight:(I)I
000170: e100 0101 |0000: shr-int/lit8 v0, v1, #int 1 // #01
000174: 0f00 |0002: return v0
</code></pre></div></div>
<p><em>(Note: output slightly trimmed)</em></p>
<p>Dalvik bytecode is register-based instead of stack-based like Java bytecode. As a result, each method only has one real bytecode which does the associated integer operation. Each uses the v1 register which will be the first argument value and an integer literal of 2 or 1.</p>
<p>So no change behavior, but D8 isn’t an optimizing compiler (although it can do <a href="/d8-optimizations/">method-local optimization</a>).</p>
<h4 id="r8">R8</h4>
<p>To run R8 we need to define a rule in order to keep our methods from being removed.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-keep,allowoptimization class ExampleKt {
<methods>;
}
</code></pre></div></div>
<p>The rules are passed with <code class="highlighter-rouge">--pg-conf</code> and we also supply the Android APIs to link against using <code class="highlighter-rouge">--lib</code>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ java -jar $R8_HOME/build/libs/r8.jar \
--lib $ANDROID_HOME/platforms/android-29/android.jar \
--release \
--pg-conf rules.txt \
--output . \
ExampleKt.class
$ dexdump -d classes.dex
Opened 'classes.dex', DEX version '035'
Class #0 -
Class descriptor : 'LExampleKt;'
Access flags : 0x0011 (PUBLIC FINAL)
Superclass : 'Ljava/lang/Object;'
Direct methods -
#0 : (in LExampleKt;)
name : 'divide'
type : '(I)I'
access : 0x0019 (PUBLIC STATIC FINAL)
code -
000118: |[000118] ExampleKt.divide:(I)I
000128: db00 0102 |0000: div-int/lit8 v0, v1, #int 2 // #02
00012c: 0f00 |0002: return v0
#1 : (in LExampleKt;)
name : 'multiply'
type : '(I)I'
access : 0x0019 (PUBLIC STATIC FINAL)
code -
000130: |[000130] ExampleKt.multiply:(I)I
000140: da00 0102 |0000: mul-int/lit8 v0, v1, #int 2 // #02
000144: 0f00 |0002: return v0
#2 : (in LExampleKt;)
name : 'shiftLeft'
type : '(I)I'
access : 0x0019 (PUBLIC STATIC FINAL)
code -
000148: |[000148] ExampleKt.shiftLeft:(I)I
000158: e000 0101 |0000: shl-int/lit8 v0, v1, #int 1 // #01
00015c: 0f00 |0002: return v0
#3 : (in LExampleKt;)
name : 'shiftRight'
type : '(I)I'
access : 0x0019 (PUBLIC STATIC FINAL)
code -
000160: |[000160] ExampleKt.shiftRight:(I)I
000170: e100 0101 |0000: shr-int/lit8 v0, v1, #int 1 // #01
000174: 0f00 |0002: return v0
</code></pre></div></div>
<p>Same exact output as D8.</p>
<h4 id="art">ART</h4>
<p>We’ll use the Dalvik bytecode output from the R8 example as the input to ART running on Android 10 on an x86 emulator.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ adb push classes.dex /sdcard/classes.dex
$ adb shell
generic_x86:/ $ su
generic_x86:/ # dex2oat --dex-file=/sdcard/classes.dex --oat-file=/sdcard/classes.oat
generic_x86:/ # oatdump --oat-file=/sdcard/classes.oat
OatDexFile:
0: LExampleKt; (offset=0x000003c0) (type_idx=1) (Initialized) (OatClassAllCompiled)
0: int ExampleKt.divide(int) (dex_method_idx=0)
CODE: (code_offset=0x00001010 size_offset=0x0000100c size=15)...
0x00001010: 89C8 mov eax, ecx
0x00001012: 8D5001 lea edx, [eax + 1]
0x00001015: 85C0 test eax, eax
0x00001017: 0F4DD0 cmovnl/ge edx, eax
0x0000101a: D1FA sar edx
0x0000101c: 89D0 mov eax, edx
0x0000101e: C3 ret
1: int ExampleKt.multiply(int) (dex_method_idx=1)
CODE: (code_offset=0x00001030 size_offset=0x0000102c size=5)...
0x00001030: D1E1 shl ecx
0x00001032: 89C8 mov eax, ecx
0x00001034: C3 ret
2: int ExampleKt.shiftLeft(int) (dex_method_idx=2)
CODE: (code_offset=0x00001030 size_offset=0x0000102c size=5)...
0x00001030: D1E1 shl ecx
0x00001032: 89C8 mov eax, ecx
0x00001034: C3 ret
3: int ExampleKt.shiftRight(int) (dex_method_idx=3)
CODE: (code_offset=0x00001040 size_offset=0x0000103c size=5)...
0x00001040: D1F9 sar ecx
0x00001042: 89C8 mov eax, ecx
0x00001044: C3 ret
</code></pre></div></div>
<p><em>(Note: output significantly trimmed)</em></p>
<p>The x86 assembly reveals that ART has indeed stepped in and normalized the arithmetic operations to use shifts!</p>
<p>First, <code class="highlighter-rouge">multiply</code> and <code class="highlighter-rouge">shiftLeft</code> now have the exact same implementation. They both use <code class="highlighter-rouge">shl</code> for a left bitwise shift of 1. Beyond this, if you look at the offsets in the file (the leftmost column), they are actually the same. ART has recognized these functions have the same body when compiled into x86 assembly and has de-duplicated them.</p>
<p>Next, while <code class="highlighter-rouge">divide</code> and <code class="highlighter-rouge">shiftRight</code> are not the same, they do share the use of <code class="highlighter-rouge">sar</code> for a right bitwise shift of 1. The four additional instructions in <code class="highlighter-rouge">divide</code> that precede <code class="highlighter-rouge">sar</code> handle the case when the input is negative by adding 1 to the value<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup>.</p>
<p>Running the same commands on a Pixel 4 running Android 10 shows how ART compiles this code to ARM assembly<sup id="fnref:2"><a href="#fn:2" class="footnote">2</a></sup>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>OatDexFile:
0: LExampleKt; (offset=0x000005a4) (type_idx=1) (Verified) (OatClassAllCompiled)
0: int ExampleKt.divide(int) (dex_method_idx=0)
CODE: (code_offset=0x00001009 size_offset=0x00001004 size=10)...
0x00001008: 0fc8 lsrs r0, r1, #31
0x0000100a: 1841 adds r1, r0, r1
0x0000100c: 1049 asrs r1, #1
0x0000100e: 4608 mov r0, r1
0x00001010: 4770 bx lr
1: int ExampleKt.multiply(int) (dex_method_idx=1)
CODE: (code_offset=0x00001021 size_offset=0x0000101c size=4)...
0x00001020: 0048 lsls r0, r1, #1
0x00001022: 4770 bx lr
2: int ExampleKt.shiftLeft(int) (dex_method_idx=2)
CODE: (code_offset=0x00001021 size_offset=0x0000101c size=4)...
0x00001020: 0048 lsls r0, r1, #1
0x00001022: 4770 bx lr
3: int ExampleKt.shiftRight(int) (dex_method_idx=3)
CODE: (code_offset=0x00001031 size_offset=0x0000102c size=4)...
0x00001030: 1048 asrs r0, r1, #1
0x00001032: 4770 bx lr
</code></pre></div></div>
<p>Once again <code class="highlighter-rouge">multiply</code> and <code class="highlighter-rouge">shiftLeft</code> both use <code class="highlighter-rouge">lsls</code> for a left shift and were de-duplicated and <code class="highlighter-rouge">shiftRight</code> uses <code class="highlighter-rouge">asrs</code> for a right shift. <code class="highlighter-rouge">divide</code> is also using <code class="highlighter-rouge">asrs</code> for its right shift, but it uses another right shift, <code class="highlighter-rouge">lsrs</code>, to handle adding 1 for negative values<sup id="fnref:3"><a href="#fn:3" class="footnote">3</a></sup>.</p>
<p>With this we can now definitively say that replacing <code class="highlighter-rouge">value * 2</code> with <code class="highlighter-rouge">value << 1</code> offers no benefit. Stop doing it for arithmetic operations and reserve it only for strictly bitwise things!</p>
<p>However, <code class="highlighter-rouge">value / 2</code> and <code class="highlighter-rouge">value >> 1</code> still produce different assembly instructions and thus presumably have different performance characteristics. Thankfully, doing <code class="highlighter-rouge">value / 2</code> avoids using generic division and is still primarily based on right shift, so they’re likely not that far apart in terms of performance.</p>
<h3 id="is-shift-faster-than-division">Is shift faster than division?</h3>
<p>To determine whether a divide or shift is faster we can use the <a href="https://developer.android.com/studio/profile/benchmark">Jetpack benchmark</a> library.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">DivideOrShiftTest</span> <span class="p">{</span>
<span class="nd">@JvmField</span> <span class="nd">@Rule</span> <span class="kd">val</span> <span class="py">benchmark</span> <span class="p">=</span> <span class="nc">BenchmarkRule</span><span class="p">()</span>
<span class="nd">@Test</span> <span class="k">fun</span> <span class="nf">divide</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">val</span> <span class="py">value</span> <span class="p">=</span> <span class="s">"4"</span><span class="p">.</span><span class="nf">toInt</span><span class="p">()</span> <span class="c1">// Ensure not a constant.</span>
<span class="kd">var</span> <span class="py">result</span> <span class="p">=</span> <span class="mi">0</span>
<span class="n">benchmark</span><span class="p">.</span><span class="nf">measureRepeated</span> <span class="p">{</span>
<span class="n">result</span> <span class="p">=</span> <span class="n">value</span> <span class="p">/</span> <span class="mi">2</span>
<span class="p">}</span>
<span class="nf">println</span><span class="p">(</span><span class="n">result</span><span class="p">)</span> <span class="c1">// Ensure D8 keeps computation.</span>
<span class="p">}</span>
<span class="nd">@Test</span> <span class="k">fun</span> <span class="nf">shift</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">val</span> <span class="py">value</span> <span class="p">=</span> <span class="s">"4"</span><span class="p">.</span><span class="nf">toInt</span><span class="p">()</span> <span class="c1">// Ensure not a constant.</span>
<span class="kd">var</span> <span class="py">result</span> <span class="p">=</span> <span class="mi">0</span>
<span class="n">benchmark</span><span class="p">.</span><span class="nf">measureRepeated</span> <span class="p">{</span>
<span class="n">result</span> <span class="p">=</span> <span class="n">value</span> <span class="n">shr</span> <span class="mi">1</span>
<span class="p">}</span>
<span class="nf">println</span><span class="p">(</span><span class="n">result</span><span class="p">)</span> <span class="c1">// Ensure D8 keeps computation.</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>I don’t have any x86 devices but I do have an ARM-based Pixel 3 running Android 10. Here are the results:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>android.studio.display.benchmark=4 ns DivideOrShiftTest.divide
count=4006
mean=4
median=4
min=4
standardDeviation=0
android.studio.display.benchmark=3 ns DivideOrShiftTest.shift
count=3943
mean=3
median=3
min=3
standardDeviation=0
</code></pre></div></div>
<p>There’s effectively zero difference between using division versus a shift with numbers this small. Those are nanoseconds, after all. Using a negative number shows no difference in the result.</p>
<p>With this we can now definitely say that replacing <code class="highlighter-rouge">value / 2</code> with <code class="highlighter-rouge">value >> 1</code> offers no benefit. Stop doing it for arithmetic operations and reserve it only for strictly bitwise things!</p>
<h3 id="can-d8r8-use-this-information-to-save-apk-size">Can D8/R8 use this information to save APK size?</h3>
<p>Given two different ways to express the same operations we should choose the one that has the better performance. But if both have the same performance, we should choose whichever results in a smaller APK size.</p>
<p>We know that <code class="highlighter-rouge">value * 2</code> and <code class="highlighter-rouge">value << 1</code> produce the same assembly from ART. Thus, if one is more space-efficient than the other in Dalvik bytecode we should unconditionally rewrite it into the smaller form. Looking at the output from D8 these produce the same size bytecode:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> #1 : (in LExampleKt;)
name : 'multiply'
⋮
000140: da00 0102 |0000: mul-int/lit8 v0, v1, #int 2 // #02
#2 : (in LExampleKt;)
name : 'shiftLeft'
⋮
000158: e000 0101 |0000: shl-int/lit8 v0, v1, #int 1 // #01
</code></pre></div></div>
<p>While there are no gains to be had for this power of 2, the multiplication runs out of bytecode space before the shift for storing the literal value. Here’s <code class="highlighter-rouge">value * 32_768</code> compared to <code class="highlighter-rouge">value << 15</code>:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> #1 : (in LExampleKt;)
name : 'multiply'
⋮
000128: 1400 0080 0000 |0000: const v0, #float 0.000000 // #00008000
00012e: 9201 0100 |0003: mul-int v1, v1, v0
#2 : (in LExampleKt;)
name : 'shiftLeft'
⋮
00015c: e000 000f |0000: shl-int/lit8 v0, v0, #int 15 // #0f
</code></pre></div></div>
<p>I have filed <a href="https://issuetracker.google.com/issues/154643053">an issue</a> on D8 to investigate optimizing this automatically, but I strongly suspect the cases where it applies to be near zero so it’s likely not worthwhile.</p>
<p>The output of D8 and R8 also tell us that <code class="highlighter-rouge">value / 2</code> and <code class="highlighter-rouge">value >> 1</code> cost the same in terms of Dalvik bytecode.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> #0 : (in LExampleKt;)
name : 'divide'
⋮
000128: db00 0102 |0000: div-int/lit8 v0, v1, #int 2 // #02
#2 : (in LExampleKt;)
name : 'shiftLeft'
⋮
000158: e000 0101 |0000: shl-int/lit8 v0, v1, #int 1 // #01
</code></pre></div></div>
<p>These will also diverge in bytecode size when the literal reaches 32,768. Unconditionally replacing a power-of-two division with a right shift is never safe because of the behavior around negatives. We could do the replacement if the value was guaranteed to be non-negative, but D8 and R8 do not track the possible ranges of integer values at this time.</p>
<h3 id="does-unsigned-number-power-of-two-division-use-shift">Does unsigned number power-of-two division use shift?</h3>
<p>Java bytecode lacks unsigned numbers, but you can emulate them by using the signed counterparts. In Java there are static helper methods for operating on signed types as unsigned values. Kotlin offers types like <code class="highlighter-rouge">UInt</code> which does similar things but completely abstracted behind a type. It’s conceivable then that when using division by a power-of-two that it could be rewritten as a shift.</p>
<p>We can use Kotlin to model both of these cases.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fun</span> <span class="nf">javaLike</span><span class="p">(</span><span class="n">value</span><span class="p">:</span> <span class="nc">Int</span><span class="p">)</span> <span class="p">=</span> <span class="nc">Integer</span><span class="p">.</span><span class="nf">divideUnsigned</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
<span class="k">fun</span> <span class="nf">kotlinLike</span><span class="p">(</span><span class="n">value</span><span class="p">:</span> <span class="nc">UInt</span><span class="p">)</span> <span class="p">=</span> <span class="n">value</span> <span class="p">/</span> <span class="mi">2U</span>
</code></pre></div></div>
<p>There’s a few cases to look at just with how the code is compiled. We’ll start with plain <code class="highlighter-rouge">kotlinc</code> (again with Kotlin 1.4-M1).</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ kotlinc Example.kt
$ javap -c ExampleKt
Compiled from "Example.kt"
public final class ExampleKt {
public static final int javaLike(int);
Code:
0: iload_0
1: iconst_2
2: invokestatic #12 // Method java/lang/Integer.divideUnsigned:(II)I
5: ireturn
public static final int kotlinLike-WZ4Q5Ns(int);
Code:
0: iload_0
1: istore_1
2: iconst_2
3: istore_2
4: iconst_0
5: istore_3
6: iload_1
7: iload_2
8: invokestatic #20 // Method kotlin/UnsignedKt."uintDivide-J1ME1BU":(II)I
11: ireturn
}
</code></pre></div></div>
<p>Kotlin does not recognize this as a power-of-two division where it could use the <code class="highlighter-rouge">iushr</code> bytecode. I’ve filed <a href="https://youtrack.jetbrains.com/issue/KT-38493">KT-38493</a> to track adding this behavior.</p>
<p>Using <code class="highlighter-rouge">-Xuse-ir</code> doesn’t change anything (except remove some of the load/store noise). However, targeting Java 8 does.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ kotlinc -jvm-target 1.8 Example.kt
$ javap -c ExampleKt
Compiled from "Example.kt"
public final class ExampleKt {
public static final int javaLike(int);
Code:
0: iload_0
1: iconst_2
2: invokestatic #12 // Method java/lang/Integer.divideUnsigned:(II)I
5: ireturn
public static final int kotlinLike-WZ4Q5Ns(int);
Code:
0: iload_0
1: iconst_2
2: invokestatic #12 // Method java/lang/Integer.divideUnsigned:(II)I
5: ireturn
}
</code></pre></div></div>
<p>The <code class="highlighter-rouge">Integer.divideUnsigned</code> method is available as of Java 8 so it’s prefered when targeting 1.8 or newer. Since this makes both function bodies identical, let’s revert back to the old output just to see what happens with it in comparison.</p>
<p>Next up is R8. Notably different from when it was invoked above is that we include the Kotlin stdlib as an input and we also pass <code class="highlighter-rouge">--min-api 24</code> since <code class="highlighter-rouge">Integer.divideUnsigned</code> is only available on API 24 and newer.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ java -jar $R8_HOME/build/libs/r8.jar \
--lib $ANDROID_HOME/platforms/android-29/android.jar \
--min-api 24 \
--release \
--pg-conf rules.txt \
--output . \
ExampleKt.class kotlin-stdlib.jar
$ dexdump -d classes.dex
Opened 'classes.dex', DEX version '039'
Class #0 -
Class descriptor : 'LExampleKt;'
Access flags : 0x0011 (PUBLIC FINAL)
Superclass : 'Ljava/lang/Object;'
Direct methods -
#0 : (in LExampleKt;)
name : 'javaLike'
type : '(I)I'
access : 0x0019 (PUBLIC STATIC FINAL)
code -
0000f8: |[0000f8] ExampleKt.javaLike:(I)I
000108: 1220 |0000: const/4 v0, #int 2 // #2
00010a: 7120 0200 0100 |0001: invoke-static {v1, v0}, Ljava/lang/Integer;.divideUnsigned:(II)I // method@0002
000110: 0a01 |0004: move-result v1
000112: 0f01 |0005: return v1
#1 : (in LExampleKt;)
name : 'kotlinLike-WZ4Q5Ns'
type : '(I)I'
access : 0x0019 (PUBLIC STATIC FINAL)
code -
000114: |[000114] ExampleKt.kotlinLike-WZ4Q5Ns:(I)I
000124: 8160 |0000: int-to-long v0, v6
000126: 1802 ffff ffff 0000 0000 |0001: const-wide v2, #double 0.000000 // #00000000ffffffff
000130: c020 |0006: and-long/2addr v0, v2
000132: 1226 |0007: const/4 v6, #int 2 // #2
000134: 8164 |0008: int-to-long v4, v6
000136: c042 |0009: and-long/2addr v2, v4
000138: be20 |000a: div-long/2addr v0, v2
00013a: 8406 |000b: long-to-int v6, v0
00013c: 0f06 |000c: return v6
</code></pre></div></div>
<p>Kotlin has its own unsigned integer division implementation which was inlined into our function. It converts the input argument and the literal to longs, performs long division, and then converts back to int. When we eventually run them through ART they’re just translated to equivalent x86 so we’re going to leave this function behind. The opportunity for optimization here was already missed.</p>
<p>For the Java version, R8 failed to replace the <code class="highlighter-rouge">divideUnsigned</code> call with a shift. I’ve filed <a href="https://issuetracker.google.com/issues/154712996">issue 154712996</a> to track this for D8 and R8.</p>
<p>The last opportunity to optimize this case is ART.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ adb push classes.dex /sdcard/classes.dex
$ adb shell
generic_x86:/ $ su
generic_x86:/ # dex2oat --dex-file=/sdcard/classes.dex --oat-file=/sdcard/classes.oat
generic_x86:/ # oatdump --oat-file=/sdcard/classes.oat
OatDexFile:
0: LExampleKt; (offset=0x000003c0) (type_idx=1) (Initialized) (OatClassAllCompiled)
0: int ExampleKt.javaLike(int) (dex_method_idx=0)
CODE: (code_offset=0x00001010 size_offset=0x0000100c size=63)...
0x00001010: 85842400E0FFFF test eax, [esp + -8192]
StackMap[0] (native_pc=0x1017, dex_pc=0x0, register_mask=0x0, stack_mask=0b)
0x00001017: 55 push ebp
0x00001018: 83EC18 sub esp, 24
0x0000101b: 890424 mov [esp], eax
0x0000101e: 6466833D0000000000 cmpw fs:[0x0], 0 ; state_and_flags
0x00001027: 0F8519000000 jnz/ne +25 (0x00001046)
0x0000102d: E800000000 call +0 (0x00001032)
0x00001032: 5D pop ebp
0x00001033: BA02000000 mov edx, 2
0x00001038: 8B85CE0F0000 mov eax, [ebp + 4046]
0x0000103e: FF5018 call [eax + 24]
StackMap[1] (native_pc=0x1041, dex_pc=0x1, register_mask=0x0, stack_mask=0b)
0x00001041: 83C418 add esp, 24
0x00001044: 5D pop ebp
0x00001045: C3 ret
0x00001046: 64FF15E0020000 call fs:[0x2e0] ; pTestSuspend
StackMap[2] (native_pc=0x104d, dex_pc=0x0, register_mask=0x0, stack_mask=0b)
0x0000104d: EBDE jmp -34 (0x0000102d)
1: int ExampleKt.kotlinLike-WZ4Q5Ns(int) (dex_method_idx=1)
CODE: (code_offset=0x00001060 size_offset=0x0000105c size=67)...
⋮
</code></pre></div></div>
<p>ART does not intrinsify calls to <code class="highlighter-rouge">divideUnsigned</code> so instead we get the machinery to jump to the regular method implementation. I filed <a href="https://issuetracker.google.com/issues/154693569">issue 154693569</a> to track adding the ART intrinsics for unsigned divide.</p>
<hr />
<p>Well that certainly was a journey. Congrats if you made it this far (or just scrolled to the bottom). Let’s summarize:</p>
<ol>
<li>ART rewrites power-of-two multiplication to left shift and power-of-two division to right shift (with a few extra instructions to handle negatives).</li>
<li>There is no observable performance difference between a right shift and power-of-two division.</li>
<li>There is no size difference in Dalvik bytecode between shifts and multiply/divide.</li>
<li>Nobody optimizes unsigned division (yet), but you’re probably not using it anyway.</li>
</ol>
<p>With these facts we can answer the title of this post:</p>
<blockquote>
<p>Which is better on Android: divide by 2 or shift by 1?</p>
</blockquote>
<p>Neither! So use division for arithmetic and only use shifts for actual bitwise operations. I’ll be switching the AndroidX collection port from shifts to multiply and divide. See you next time.</p>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>-3 in binary is 0b11111101. If we attempt to divide by 2 by solely performing the right shift the result is 0b11111110 which is -2, an incorrect result. By adding 1 to -3 first we get -2 which in binary is 0b11111110. Shifted right we get 0b11111111 which is -1, the correct result.</p>
<p>In terms of the actual instructions:</p>
<ul>
<li><code class="highlighter-rouge">mov eax, ecx</code> saves the original input argument value in <code class="highlighter-rouge">eax</code>.</li>
<li><code class="highlighter-rouge">lea edx, [eax + 1]</code> adds 1 to the input argument and stores the result in <code class="highlighter-rouge">edx</code>, the register we will be shifting.</li>
<li><code class="highlighter-rouge">test eax, eax</code> does a bitwise AND of the input argument against itself which results in a few registers being set based on properties of the input argument.</li>
<li><code class="highlighter-rouge">cmovnl/ge edx, eax</code> then maybe overwrites <code class="highlighter-rouge">edx</code> (value+1) with <code class="highlighter-rouge">eax</code> (value) based on the result of the <code class="highlighter-rouge">test</code>.</li>
</ul>
<p>From there the instructions do a normal right shift. This is basically equivalent to <code class="highlighter-rouge">(value < 0 ? value + 1 : value) >> 1</code>. <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
<li id="fn:2">
<p>Thanks to <a href="https://twitter.com/ZelenetS">Sergey Vasilinets</a> for providing this. <code class="highlighter-rouge">dex2oat</code> can only be run as root on modern Android versions so a normal Android install such as on my Pixel 3 can’t run it. <a href="#fnref:2" class="reversefootnote">↩</a></p>
</li>
<li id="fn:3">
<p>In terms of the actual instructions:</p>
<ul>
<li><code class="highlighter-rouge">lsrs r0, r1, #31</code> does a logical (i.e., not sign-extending) shift of the input argument by 31 bits into <code class="highlighter-rouge">r0</code>. This results in 1 for negative numbers and 0 for positive numbers.</li>
<li><code class="highlighter-rouge">adds r1, r0, r1</code> adds the result of the previous instruction to the input argument, effectively adding 1 to negative inputs.</li>
</ul>
<p>From there the instructions do a normal right shift. This is basically equivalent to <code class="highlighter-rouge">(value + (value >>> 31)) >> 1</code>. <a href="#fnref:3" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
Simple Multiplatform RPC with Kotlin Serialization2020-04-15T00:00:00+00:00https://jakewharton.com/simple-multiplatform-rpc-with-kotlin-serialization<p>I recently played a minor role in helping add Cast support to an Android app. Both the Android app and Cast display are written in Kotlin. The Android Cast SDK relays JSON strings to the JavaScript SDK which invokes your callback with the deserialized equivalent as a JS object. A multiplatform library holds the model objects so that they can be shared between Android and JS.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Game</span><span class="p">(</span>
<span class="kd">val</span> <span class="py">players</span><span class="p">:</span> <span class="nc">Array</span><span class="p"><</span><span class="nc">Player</span><span class="p">></span>
<span class="p">)</span>
<span class="kd">class</span> <span class="nc">Player</span><span class="p">(</span>
<span class="kd">val</span> <span class="py">name</span><span class="p">:</span> <span class="nc">String</span><span class="p">,</span>
<span class="kd">val</span> <span class="py">color</span><span class="p">:</span> <span class="nc">String</span><span class="p">,</span>
<span class="kd">val</span> <span class="py">scores</span><span class="p">:</span> <span class="nc">Array</span><span class="p"><</span><span class="nc">Int</span><span class="p">></span>
<span class="p">)</span>
</code></pre></div></div>
<p><a href="https://github.com/square/moshi/">Moshi</a> serializes the models to JSON in the Android app.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">val</span> <span class="py">game</span> <span class="p">=</span> <span class="nc">Game</span><span class="p">(</span><span class="nf">arrayOf</span><span class="p">(</span>
<span class="nc">Player</span><span class="p">(</span><span class="s">"Jesse"</span><span class="p">,</span> <span class="s">"#ff0000"</span><span class="p">,</span> <span class="nf">arrayOf</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">)),</span>
<span class="nc">Player</span><span class="p">(</span><span class="s">"Matt"</span><span class="p">,</span> <span class="s">"#ff00ff"</span><span class="p">,</span> <span class="nf">arrayOf</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="p">))</span>
<span class="p">))</span>
<span class="kd">val</span> <span class="py">gameAdapter</span> <span class="p">=</span> <span class="n">moshi</span><span class="p">.</span><span class="nf">adapter</span><span class="p">(</span><span class="nc">Game</span><span class="o">::</span><span class="k">class</span><span class="p">.</span><span class="n">java</span><span class="p">)</span>
<span class="kd">val</span> <span class="py">gameJson</span> <span class="p">=</span> <span class="n">gameAdapter</span><span class="p">.</span><span class="nf">toJson</span><span class="p">(</span><span class="n">game</span><span class="p">)</span>
<span class="c1">// {"players":[{"name":"Jesse",...},{"name":"Matt",...}]}</span>
<span class="n">castSdk</span><span class="p">.</span><span class="nf">send</span><span class="p">(</span><span class="n">gameJson</span><span class="p">)</span>
</code></pre></div></div>
<p>The Cast app receives the deserialized JS object and interprets it as being of the same type.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">castSdk</span><span class="p">.</span><span class="nf">addCustomMessageListener</span> <span class="p">{</span> <span class="n">message</span> <span class="p">-></span>
<span class="kd">val</span> <span class="py">game</span> <span class="p">=</span> <span class="n">message</span><span class="p">.</span><span class="n">data</span><span class="p">.</span><span class="n">unsafeCast</span><span class="p"><</span><span class="nc">Game</span><span class="p">>()</span>
<span class="n">ui</span><span class="p">.</span><span class="nf">render</span><span class="p">(</span><span class="n">game</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This works but imposes some severe limitations. The model objects can only use collections available natively to JS which means <code class="highlighter-rouge">Array</code>s instead of <code class="highlighter-rouge">List</code>s. Custom serialization is also not supported because the JSON to JS object conversion was happening outside the library.</p>
<p>It was clear this setup wasn’t going to work long-term.</p>
<h3 id="kotlin-serialization">Kotlin Serialization</h3>
<p><a href="https://github.com/Kotlin/kotlinx.serialization">kotlinx.serialization</a> is Kotlin’s multiplatform, reflection-free, format-agnostic serialization library. Its compiler plugin generates code for types which are annotated as <code class="highlighter-rouge">@Serializable</code>.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gi">+@Serializable
</span> class Game(
val players: Array<Player>
)
<span class="gi">+@Serializable
</span> class Player(
val name: String,
</code></pre></div></div>
<p>Updating the Android app requires specifying that we’re using the JSON format and supplying a reference to the generated serializer.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gd">-val gameAdapter = moshi.adapter(Game::class.java)
-val gameJson = gameAdapter.toJson(game)
</span><span class="gi">+val gameJson = Json.stringify(Game.serializer(), game)
</span> // {"players":[{"name":"Jesse",...},{"name":"Matt",...}]}
castSdk.send(gameJson)
</code></pre></div></div>
<p>Normally in this situation, changing the serialization library would only affect the Android app since the Cast SDK internally parses JSON to JS objects. However, kotlinx.serialization has the unique feature of being able to “parse” a JS object.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gi">+val objectParser = DynamicObjectParser()
</span> castSdk.addCustomMessageListener { message ->
<span class="gd">- val game = message.data.unsafeCast<Game>()
</span><span class="gi">+ val game = objectParser.parse(message.data, Game.serializer())
</span> ui.render(game)
}
</code></pre></div></div>
<p>This walks the object properties as if it were JSON and passes them through the serializer. Now we can use all of the features of the library from custom serializers to simple things like using a <code class="highlighter-rouge">List</code>.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> @Serializable
class Game(
<span class="gd">- val players: Array<Player>
</span><span class="gi">+ val players: List<Player>
</span> )
@Serializable
class Player(
val name: String,
val color: String,
<span class="gd">- val scores: Array<Int>
</span><span class="gi">+ val scores: List<Int>
</span> )
</code></pre></div></div>
<p>This future-proofed the app to ensure that its models could continue to be shared even as they grew in complexity. And they were about to.</p>
<h3 id="simple-rpcs">Simple RPCs</h3>
<p>The Cast app started as a stateless rendering of the game model but it lacked some of the Android app’s flair. Instead of sending only the bare model, the Android app was changed to send an event. This allowed showing animations on the Cast display after an action. Each event contained a copy of the game model as well as any other information about the event.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@Serializable</span>
<span class="kd">data class</span> <span class="nc">PlayerAdded</span><span class="p">(</span>
<span class="kd">val</span> <span class="py">game</span><span class="p">:</span> <span class="nc">Game</span><span class="p">,</span>
<span class="kd">val</span> <span class="py">player</span><span class="p">:</span> <span class="nc">Player</span>
<span class="p">)</span>
<span class="nd">@Serializable</span>
<span class="kd">data class</span> <span class="nc">SpinTheBottle</span><span class="p">(</span>
<span class="kd">val</span> <span class="py">game</span><span class="p">:</span> <span class="nc">Game</span><span class="p">,</span>
<span class="kd">val</span> <span class="py">winner</span><span class="p">:</span> <span class="nc">Int</span>
<span class="p">)</span>
</code></pre></div></div>
<p>The type will determine the behavior of the Cast app in response to these events.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">when</span> <span class="p">(</span><span class="n">event</span><span class="p">)</span> <span class="p">{</span>
<span class="k">is</span> <span class="nc">PlayerAdded</span> <span class="p">-></span> <span class="p">{</span> <span class="o">..</span> <span class="p">}</span>
<span class="k">is</span> <span class="nc">SpinTheBottle</span> <span class="p">-></span> <span class="p">{</span> <span class="o">..</span> <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Unfortunately this does not work as-is. When serialized, the root JSON object contains only the properties of the object and not which specific type was serialized.</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="dl">"</span><span class="s2">game</span><span class="dl">"</span><span class="p">:{</span> <span class="cm">/*..*/</span> <span class="p">},</span><span class="dl">"</span><span class="s2">winner</span><span class="dl">"</span><span class="p">:</span><span class="mi">1</span><span class="p">}</span>
</code></pre></div></div>
<p>You can try to infer the type from which properties are present but it’s a brittle setup.</p>
<p>This is generally solved by using something called “polymorphic serialization” which uses some kind of marker to encode which type was serialized. In kotlinx.serialization 0.14.0, the compiler automatically enables polymorphic serialization for Kotlin sealed hierarchies so it’s an obvious choice.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gi">+@Serializable
+sealed class GameEvent {
+ abstract val game: Game
+}
</span>
@Serializable
data class PlayerAdded(
<span class="gd">- val game: Game,
</span><span class="gi">+ override val game: Game,
</span> val player: Player
<span class="gd">-)
</span><span class="gi">+) : GameEvent()
</span>
@Serializable
data class SpinTheBottle(
<span class="gd">- val game: Game,
</span><span class="gi">+ override val game: Game,
</span> val winner: Int
<span class="gd">-)
</span><span class="gi">+) : GameEvent()
</span></code></pre></div></div>
<p>The JSON will now include a discriminator, a string identifying which type was used, so that the deserialization code picks the corresponding type on the other side. By default the library uses array-based discriminators (but you could elect to add a property to the object itself).</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="dl">"</span><span class="s2">com.example.model.SpinTheBottle</span><span class="dl">"</span><span class="p">,{</span><span class="dl">"</span><span class="s2">game</span><span class="dl">"</span><span class="p">:{</span> <span class="cm">/*..*/</span> <span class="p">},</span><span class="dl">"</span><span class="s2">winner</span><span class="dl">"</span><span class="p">:</span><span class="mi">1</span><span class="p">}]</span>
</code></pre></div></div>
<p>Additionally, by using a sealed class, Kotlin can now enforce that a <code class="highlighter-rouge">when</code> on the event types is exhaustive<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup>.</p>
<p>kotlinx.serialization 0.20.0 added support for polymorphic serialization in <code class="highlighter-rouge">DynamicObjectParser</code> allowing the Cast app to take advantage of it.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> val objectParser = DynamicObjectParser()
castSdk.addCustomMessageListener { message ->
<span class="gd">- val game = objectParser.parse(message.data, Game.serializer())
</span><span class="gi">+ val event = objectParser.parse(message.data, GameEvent.serializer())
+ val game = event.game
</span> ui.render(game)
<span class="gi">+ when (event) {
+ is PlayerAdded -> { .. }
+ is SpinTheBottle -> { .. }
+ }
</span> }
</code></pre></div></div>
<p>This setup creates a pretty robust unidirectional RPC system for the Android app to talk to the Cast display. The build will fail if you forget to handle a new event on the Cast side. The sending code and transport don’t need updated for new events since it’s all based on the <code class="highlighter-rouge">GameEvent</code> supertype.</p>
<hr />
<p>With the Cast SDK imposing JSON and automatic deserialization to JS objects, the feature set of Kotlin serialization fits right in. It allows maximizing code reuse without imposing too much complexity. And, granted, it’s just about the most basic RPC system you could build, but it serves the app well. Supporting requirements like associated responses and bidirectional streaming is better left to more heavyweight systems like gRPC.</p>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>Note: The snippet with this code is not set up to be exhaustive for simplicity. <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
Litmus-Testing Kotlin's Many Memory Models2020-04-08T00:00:00+00:00https://jakewharton.com/litmus-testing-kotlins-many-memory-models<p>When writing multiplatform code, Kotlin’s three compiler backends each have different memory models which must be considered.</p>
<p>JavaScript is single-threaded so you really can do no wrong. The JVM model is arguably too permissive where you can do incorrect things and have them work 99.9% of the time. When targeting native, Kotlin enforces some invariants which helps prevent you from those 0.1% bugs that crop up in the JVM.</p>
<p>I’ve been porting the <a href="https://developer.android.com/reference/androidx/collection/package-summary">AndroidX collection library</a> to Kotlin multiplatform to experiment with binary compatibility, performance, tooling, and the different memory models. The library consists of mutable, single-threaded data structures. This should mean the different memory models never come into play. But weirdly they do, and let’s look at how.</p>
<h3 id="on-deck">On Deck</h3>
<p>The Kotlin standard library contains general-purpose collections like lists, sets, and maps in both mutable and read-only form. Kotlin 1.3.70 added another collection, <a href="https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.collections/-array-deque/"><code class="highlighter-rouge">ArrayDeque</code></a>, a “double-ended queue” for efficient stacks and queues.</p>
<p>During the 1.3.70 EAP, Kevin Galligan opened <a href="https://github.com/JetBrains/kotlin-native/issues/3876">an issue</a> where <code class="highlighter-rouge">ArrayDeque</code> could <em>only</em> be instantiated on the main thread and not a background thread when targeting Kotlin/Native. At the time I didn’t read into it, but as I was porting these collections it came to mind.</p>
<p>The underlying cause was that the implementation relied on a top-level <code class="highlighter-rouge">val</code> for a shared, empty array when the collection was empty. Arrays are fixed-length, so an empty array is effectively immutable and thus can be shared by all empty collections. But that seems fine?</p>
<p>It <em>is</em> fine for Kotlin/JS and Kotlin/JVM but Kotlin/Native is different here. By default, Kotlin/Native only allows the main thread to access top-level <code class="highlighter-rouge">val</code>s. If you want to access the value from multiple threads (potentially concurrently) you must choose whether you want thread-local or shared-but-immutable behavior with an annotation. <code class="highlighter-rouge">ArrayDeque</code>’s empty array was missing this annotation.</p>
<p>As it turns out, my collections had the exact same issue! Each started with a shared, empty array and only allocated its own storage when the first element arrived. I had tests, but the tests were only exercising the type on the main thread. It’s an easy fix, just add <code class="highlighter-rouge">@SharedImmutable</code>, but how do I prevent regression and future problems of this nature?</p>
<h3 id="testing-threads">Testing Threads</h3>
<p>Since Kotlin/Native enforces different semantics between its main thread and background threads, it’s only logical to run the tests once on the main thread and once on a background thread to ensure compliance.</p>
<p>If our test is written solely for Kotlin/Native this is pretty easy. The native version of the standard library has a <code class="highlighter-rouge">Worker</code> API for running on a background thread.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fun</span> <span class="nf">threadedTest</span><span class="p">(</span><span class="n">body</span><span class="p">:</span> <span class="p">()</span> <span class="p">-></span> <span class="nc">Unit</span><span class="p">)</span> <span class="p">{</span>
<span class="nf">body</span><span class="p">()</span>
<span class="n">body</span><span class="p">.</span><span class="nf">freeze</span><span class="p">()</span>
<span class="kd">val</span> <span class="py">worker</span> <span class="p">=</span> <span class="nc">Worker</span><span class="p">.</span><span class="nf">start</span><span class="p">()</span>
<span class="kd">val</span> <span class="py">future</span> <span class="p">=</span> <span class="n">worker</span><span class="p">.</span><span class="nf">execute</span><span class="p">(</span><span class="nc">SAFE</span><span class="p">,</span> <span class="p">{</span> <span class="n">body</span> <span class="p">})</span> <span class="p">{</span>
<span class="nf">runCatching</span><span class="p">(</span><span class="n">it</span><span class="p">)</span>
<span class="p">}</span>
<span class="n">future</span><span class="p">.</span><span class="n">result</span><span class="p">.</span><span class="nf">getOrThrow</span><span class="p">()</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This function accepts a lambda which it runs synchronously (which will be on the main thread) and then transfers that lambda to a background thread where it’s run a second time. The main thread blocks on the result of the background thread where it rethrows any exceptions that occurred.</p>
<p>Each test case is updated to put its body inside a call to this function.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gd">-@Test fun isEmpty() {
</span><span class="gi">+@Test fun isEmpty() = threadedTest {
</span> val map = ArrayMap()
assertTrue(map.isEmpty())
}
</code></pre></div></div>
<p>Running without <code class="highlighter-rouge">@SharedImmutable</code> now causes the test to correctly fail. Say goodbye to an entire class of Kotlin/Native bugs!</p>
<h3 id="multiplatform">Multiplatform</h3>
<p>For multiplatform libraries, like my collection library, the tests are written in platform-agnostic “common” Kotlin with no access to the Kotlin/Native-specific <code class="highlighter-rouge">Worker</code> API. We can instead rely on the expect/actual language feature of multiplatform Kotlin to make this work.</p>
<p>In <code class="highlighter-rouge">src/commonTest/kotlin/</code> the <code class="highlighter-rouge">threadedTest</code> function is declared as an <code class="highlighter-rouge">expect fun</code>:</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">expect</span> <span class="k">fun</span> <span class="nf">threadedTest</span><span class="p">(</span><span class="n">body</span><span class="p">:</span> <span class="p">()</span> <span class="p">-></span> <span class="nc">Unit</span><span class="p">)</span>
</code></pre></div></div>
<p>The native-specific implementation is put in <code class="highlighter-rouge">src/nativeTest/kotlin/</code>:</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">actual</span> <span class="k">fun</span> <span class="nf">threadedTest</span><span class="p">(</span><span class="n">body</span><span class="p">:</span> <span class="p">()</span> <span class="p">-></span> <span class="nc">Unit</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// Same as Kotlin/Native code from previous section.</span>
<span class="p">}</span>
</code></pre></div></div>
<p>For JavaScript in <code class="highlighter-rouge">src/jsTest/kotlin/</code> we don’t need threading so its implementation just inlines itself away.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">actual</span> <span class="k">inline</span> <span class="k">fun</span> <span class="nf">threadedTest</span><span class="p">(</span><span class="n">body</span><span class="p">:</span> <span class="p">()</span> <span class="p">-></span> <span class="nc">Unit</span><span class="p">)</span> <span class="p">=</span> <span class="nf">body</span><span class="p">()</span>
</code></pre></div></div>
<p>For the JVM in <code class="highlighter-rouge">src/jvmTest/kotlin/</code> you’re free to either inline it away like JavaScript or use the <code class="highlighter-rouge">Thread</code> APIs to invoke <code class="highlighter-rouge">body</code> twice. Since the memory models of the JVM and Android give no special treatment to the main thread there’s really no reason to run it twice.</p>
<p>Now our test from the previous section can live in <code class="highlighter-rouge">src/commonTest/kotlin/</code> and wrap itself in <code class="highlighter-rouge">threadedTest</code>. On JS and JVM the test will run normally and only on native targets will it run twice.</p>
<hr />
<p>The memory model of Kotlin/Native helps eliminate bugs that would probabilistically occur on more permissive platforms like the JVM. With the constraints of its memory model being runtime checked, running your unit tests on both the main thread and a background thread prevent bugs like the one which occurred with <code class="highlighter-rouge">ArrayDeque</code>.</p>
<p>I filed <a href="https://github.com/JetBrains/kotlin-native/issues/4075">an issue</a> on the Kotlin/Native repo asking for some kind of built-in mechanism to support this use case. And ideally it would be something that you could apply to a whole class rather than having to remember to do it for each function.</p>
D8 Optimization: Assertions2020-03-25T00:00:00+00:00https://jakewharton.com/d8-optimization-assertions<blockquote>
<p>Note: This post is part of a series on D8 and R8, Android’s new dexer and optimizer, respectively. For an intro to D8 read <a href="/androids-java-8-support/">“Android’s Java 8 support”</a>. For an intro to R8 read <a href="/r8-optimization-staticization/">“R8 Optimization: Staticization”</a>.</p>
</blockquote>
<p>The <code class="highlighter-rouge">assert</code> keyword is quirky Java language syntax used for testing invariants. That is: things you expect to <strong>always</strong> be true.</p>
<p>Its syntax has two forms:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">assert</span> <span class="o"><</span><span class="n">bool</span><span class="o">-</span><span class="n">expression</span><span class="o">>;</span>
<span class="k">assert</span> <span class="o"><</span><span class="n">bool</span><span class="o">-</span><span class="n">expression</span><span class="o">></span> <span class="o">:</span> <span class="o"><</span><span class="n">expression</span><span class="o">>;</span>
</code></pre></div></div>
<p>The first expression will only be evaluated at runtime if the <code class="highlighter-rouge">-ea</code> (enable assertions) flag is set on the JVM. The second expression, if present, is used as the argument to the <code class="highlighter-rouge">AssertionError</code> constructor that’s thrown if the first expression returns false.</p>
<p>As an Android developer you might not be too familiar with <code class="highlighter-rouge">assert</code>. This is because every Android app runs on a VM which is forked from a shared “zygote” process which has assertions disabled. Thus, even if you put an <code class="highlighter-rouge">assert</code> in your code, there is no way to actually enable it.</p>
<p>So why bother talking about it? Well it turns out they’re about to become useful on Android for the first time!</p>
<h3 id="todays-behavior">Today’s behavior</h3>
<p><code class="highlighter-rouge">assert</code> statements guard things which must <em>always</em> be true in order for your program to execute correctly. Let’s write one.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">IdGenerator</span> <span class="o">{</span>
<span class="kd">private</span> <span class="kt">int</span> <span class="n">id</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span>
<span class="kt">int</span> <span class="nf">next</span><span class="o">()</span> <span class="o">{</span>
<span class="k">assert</span> <span class="nc">Thread</span><span class="o">.</span><span class="na">currentThread</span><span class="o">()</span> <span class="o">==</span> <span class="nc">Looper</span><span class="o">.</span><span class="na">getMainLooper</span><span class="o">().</span><span class="na">getThread</span><span class="o">();</span>
<span class="k">return</span> <span class="n">id</span><span class="o">++;</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>This class creates unique IDs and guarantees they’re unique by only allowing calls from the main thread. If this class was called concurrently from multiple threads you might see duplicate values. Sure it’s a little contrived and there’s things like <code class="highlighter-rouge">@MainThread</code> which is checked by Lint but we’re focusing on <code class="highlighter-rouge">assert</code> so roll with it.</p>
<p>The <a href="/r8-optimization-null-data-flow-analysis-part-1/">Null Data Flow Analysis</a> post introduced the SSA form that R8 uses to eliminate branches of code which it can prove will never be executed. The SSA for the <code class="highlighter-rouge">next()</code> method when parsed from Java bytecode looks <em>very</em> roughly like this:</p>
<!--
digraph G {
rankdir="RL";
"if thread == main thread" -> "if assertions enabled"
"throw AssertionError" -> "if thread == main thread"
"int value = id" -> "if thread == main thread"
"int value = id" -> "if assertions enabled"
"id = value + 1" -> "int value = id"
"return value" -> "id = value + 1"
}
-->
<p><a href="/static/post-image/assert-1.png"><img src="/static/post-image/assert-1.png" /></a></p>
<p>D8 knows that Android does not support Java assertions. It will remove the check and replace it with <code class="highlighter-rouge">false</code> allowing dead-code elimination to occur. This propagates to the nodes which can only be taken when it returns true.</p>
<!--
digraph G {
rankdir="RL";
"if assertions enabled" [style=dotted]
"if thread == main thread"[style=dotted]
"throw AssertionError" [style=dotted]
"if thread == main thread" -> "if assertions enabled" [style=dotted]
"throw AssertionError" -> "if thread == main thread" [style=dotted]
"int value = id" -> "if thread == main thread" [style=dotted]
"int value = id" -> "if assertions enabled" [style=dotted]
"id = value + 1" -> "int value = id"
"return value" -> "id = value + 1"
}
-->
<p><a href="/static/post-image/assert-2.png"><img src="/static/post-image/assert-2.png" /></a></p>
<p>As a result, the boolean expression and optional message expression are entirely eliminated from the bytecode. Only the field read, field increment, and return remain.</p>
<!--
digraph G {
rankdir="RL";
"id = value + 1" -> "int value = id"
"return value" -> "id = value + 1"
}
-->
<p><a href="/static/post-image/assert-3.png"><img src="/static/post-image/assert-3.png" style="max-height: 43px" /></a></p>
<p>We can confirm this by sending the Java source through the compilation pipeline:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>javac <span class="nt">-bootclasspath</span> <span class="nv">$ANDROID_HOME</span>/platforms/android-29/android.jar IdGenerator.java
<span class="nv">$ </span>java <span class="nt">-jar</span> <span class="nv">$R8_HOME</span>/build/libs/d8.jar <span class="se">\</span>
<span class="nt">--lib</span> <span class="nv">$ANDROID_HOME</span>/platforms/android-29/android.jar <span class="se">\</span>
<span class="nt">--output</span> <span class="nb">.</span> <span class="se">\</span>
IdGenerator.class
<span class="nv">$ </span>dexdump <span class="nt">-d</span> classes.dex
⋮
<span class="o">[</span>00011c] IdGenerator.next:<span class="o">()</span>I
0000: iget v0, v2, LIdGenerator<span class="p">;</span>.id:I
0002: add-int/lit8 v1, v0, <span class="c">#int 1</span>
0004: iput v1, v2, LIdGenerator<span class="p">;</span>.id:I
0006: <span class="k">return </span>v0
⋮
</code></pre></div></div>
<p>Eliminating a runtime check which always returns false is an easy win, but the SSA form means that we eliminate the bytecode for both expressions of the <code class="highlighter-rouge">assert</code> statement including any intermediate values they rely on.</p>
<h3 id="tomorrows-behavior">Tomorrow’s behavior</h3>
<p>The version of D8 in AGP 4.1 slightly changes the thinking around Java <code class="highlighter-rouge">assert</code>. Instead of assuming that the runtime check will always fail at runtime (which it <em>still</em> does), it computes the check at compile-time based on whether your build is debuggable.</p>
<p>In practice, this means that any debug variant will replace the assertions-enabled check at compile-time with <code class="highlighter-rouge">true</code>.</p>
<!--
digraph G {
rankdir="RL";
"if assertions enabled" [style=dotted]
"if thread == main thread" -> "if assertions enabled" [style=dotted]
"throw AssertionError" -> "if thread == main thread"
"int value = id" -> "if thread == main thread"
"int value = id" -> "if assertions enabled" [style=dotted]
"id = value + 1" -> "int value = id"
"return value" -> "id = value + 1"
}
-->
<p><a href="/static/post-image/assert-4.png"><img src="/static/post-image/assert-4.png" /></a></p>
<p>This eliminates the enabled check but retains the invariant check.</p>
<!--
digraph G {
rankdir="RL";
"throw AssertionError" -> "if thread == main thread"
"int value = id" -> "if thread == main thread"
"id = value + 1" -> "int value = id"
"return value" -> "id = value + 1"
}
-->
<p><a href="/static/post-image/assert-5.png"><img src="/static/post-image/assert-5.png" style="max-height:97px" /></a></p>
<p>Sending <code class="highlighter-rouge">IdGenerator</code> through D8 with the <code class="highlighter-rouge">--force-enable-assertions</code> flag that AGP automatically adds for debug variants shows this in Dalvik bytecode:</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> $ java -jar $R8_HOME/r8/build/libs/d8.jar \
--lib $ANDROID_HOME/platforms/android-29/android.jar \
<span class="gi">+ --force-enable-assertions \
</span> --output . \
IdGenerator.class
$ dexdump -d classes.dex
⋮
[000190] IdGenerator.next:()I
<span class="gi">+0000: invoke-static {}, Ljava/lang/Thread;.currentThread:()Ljava/lang/Thread;
+0003: move-result-object v0
+0004: invoke-static {}, Landroid/os/Looper;.getMainLooper:()Landroid/os/Looper;
+0007: move-result-object v1
+0008: invoke-virtual {v1}, Landroid/os/Looper;.getThread:()Ljava/lang/Thread;
+000b: move-result-object v1
+000c: if-ne v0, v1, 0015
</span> 000e: iget v0, v2, LIdGenerator;.id:I
0010: add-int/lit8 v1, v0, #int 1
0012: iput v1, v2, LIdGenerator;.id:I
0014: return v0
<span class="gi">+0015: new-instance v0, Ljava/lang/AssertionError;
+0017: invoke-direct {v0, v1}, Ljava/lang/AssertionError;.<init>:()V
+001a: throw v0
</span> ⋮
</code></pre></div></div>
<p>Our debug build still tests the invariant at runtime but the release build completely eliminates the check. This behavior is now similar to the JVM where unit tests turn on the <code class="highlighter-rouge">-ea</code> flag whereas production does not.</p>
<p>(If you’re wondering why the code which throws the exception was moved to the bottom of the method check out the <a href="/optimizing-bytecode-by-manipulating-source-code/">Optimizing Bytecode by Manipulating Source Code</a> post.)</p>
<hr />
<p>This feature is already available in the latest AGP 4.1 alphas. The nature of invariants are such that they should never fail unless you’re already doing something very wrong. By checking them in debug builds we have only confidence to gain in the correctness of our libraries and application code when running on Android.</p>
<p>Kotlin’s <code class="highlighter-rouge">assert()</code> function currently has a subtle behavior difference compared to Java’s <code class="highlighter-rouge">assert</code> keyword. For more information see Jesse Wilson’s <a href="https://publicobject.com/2019/11/18/kotlins-assert-is-not-like-javas-assert/">Kotlin’s Assert Is Not Like Java’s Assert</a> post. D8 currently does not recognize Kotlin’s <code class="highlighter-rouge">assert()</code> to apply the optimization in this post, but the <a href="https://issuetracker.google.com/issues/139898386">original D8 feature request</a> remains open for this very reason.</p>
<p>Unlike some of the R8 optimizations covered in recent posts, this optimization is localized to the body of a single method which is why it can also be performed by D8. Check out the <a href="/d8-optimizations/">D8 Optimizations</a> post for more optimizations which apply in both D8 and R8.</p>
<p>And stay tuned for more D8 and R8 optimization posts coming soon!</p>
Removing Google as a Single Point of Failure Part 2: Gmail2020-03-18T00:00:00+00:00https://jakewharton.com/removing-google-as-a-single-point-of-failure-gmail<p>I want to remove Google as a single point of failure in my life. In <a href="/removing-google-as-a-single-point-of-failure/">the first blog post</a> on this subject I detailed my setup for backing up Google Photos and Google Drive contents onto my home server and remotely to <a href="https://rsync.net">rsync.net</a>. Left out of that post was a solution for Gmail because I hadn’t found one yet. Now I have.</p>
<h3 id="source-of-truth">Source of truth</h3>
<p>That first post started with an important qualification:</p>
<blockquote>
<p>This does not mean that I’m going to stop using Google products. Quite the opposite. Gmail, Google Photos, and Google Drive will remain the source-of-truth for all of the things I listed above. What’s different is that should Google disappear tomorrow (or just my account) I would lose no data.</p>
</blockquote>
<p>This was easy to achieve with Photos and Drive because the data is all there is. With email that’s unfortunately not true.</p>
<p>Incrementally backing up the email data is pretty straightforward–we’ll get into that shortly. But with Gmail your email address is still tied to the <code class="highlighter-rouge">@gmail.com</code> domain. So if my account or all of Google disappears, I won’t be able to receive any more email.</p>
<p>Of course the “easy” fix here is to just use a domain that I control. Obviously I own <code class="highlighter-rouge">jakewharton.com</code>, and I intend to set that up, but I wanted something shorter. I’ve owned <code class="highlighter-rouge">cob.io</code> for many years with the intention of setting up <code class="highlighter-rouge">j@cob.io</code>, but I go by “Jake”. Luckily the last few years have seen an influx of new TLDs so I managed to grab <code class="highlighter-rouge">ke.fyi</code>. Say hello to <code class="highlighter-rouge">j@ke.fyi</code>!</p>
<p>Having an email on my own domain doesn’t address the problem that there’s still hundreds or thousands of services that I’ve given the Gmail address to. While I can migrate many, there are inevitably those which I can’t or that I simply don’t know exist. The old address needs to remain working.</p>
<h3 id="fastmail">Fastmail</h3>
<p>After browsing a few hosted email solutions, I settled on <a href="https://ref.fm/u23361320">Fastmail</a> <em>(Note: referral link)</em>. In addition to a positive recommendation from a friend, there were a few key motivating factors.</p>
<h4 id="domain-catch-all">Domain catch-all</h4>
<p>A popular feature of Gmail is the ability to append a <code class="highlighter-rouge">+</code> to your user followed by any text and mail will still be sent to you. This can be used for filters or to see who is selling your email address to others.</p>
<p>A domain catch-all is the same thing but you can change the entire username. Now I can use addresses like <code class="highlighter-rouge">southwest@ke.fyi</code> without needing to set anything up first. Aside from knowing if they sell my email it also slightly improves security. While the format is human guessable, any automated attack using emails from a data breach simply don’t exist on other services.</p>
<p>Fastmail supports replying to these catch-all emails using the same address to which it was sent. This is critical to maintain the illusion, especially when dealing with people rather than automated systems.</p>
<h4 id="multiple-domains">Multiple domains</h4>
<p>Aside from <code class="highlighter-rouge">ke.com</code> I also set up <code class="highlighter-rouge">jakewharton.com</code> and a few other domains. Fastmail sends all emails to my configured domains to a unified inbox rather than forcing me to switch accounts. Instead, my replies will match the incoming address the same as it did for the catch-all.</p>
<p>Additionally, when composing emails I can choose the domain from which it will be sent. And for those with catch-all set up, I can even pick arbitrary usernames on those domains. Neat!</p>
<h4 id="gmail-support">Gmail support</h4>
<p>Since my Gmail address will receive <em>some</em> mail for the foreseeable future it’s important to use a service that supports more than just a one-time import. Fastmail performs near-realtime incremental syncs to pull in any new email or calendar events from Gmail. Not only is it very fast, but they seem to be able to bypass the rate limits that otherwise exist when downloading your email over IMAP.</p>
<p>I can compose email using the Gmail address. I don’t know why I would ever need this, but it’s nice to have.</p>
<p>In replies to any email Fastmail lets me change the address from which I’m replying. When a person sends an email to my old address, I can use this feature to gradually migrate them over to the new one.</p>
<h4 id="imap-availability">IMAP availability</h4>
<p>Remember, it’s not enough to migrate from Gmail to Fastmail for an address on our own domain. We still need to ensure a backup solution. Thankfully Fastmail supports any and every protocol you’d need.</p>
<p>As a nice bonus, the Gmail to Fastmail sync bypasses Google’s rate limit meaning you can also sync your entire Gmail within minutes through Fastmail rather than having to spread it out over multiple days when accessing it directly.</p>
<h3 id="backup">Backup</h3>
<p>Almost immediately after the previous blog post people were sending me a myriad of tools for Gmail backup. Thank you for that!</p>
<h4 id="mbsync">mbsync</h4>
<p>After trying a few tools I settled on <code class="highlighter-rouge">mbsync</code> which is part of the <a href="http://isync.sourceforge.net/">isync</a> project. The tool is very generic but can be used to synchronize emails to the <a href="https://en.wikipedia.org/wiki/Maildir">Maildir</a> format over IMAP.</p>
<p>Maildir is a standard format that can be read by many tools. Unlike mbox, the format used in Google Takeout for Gmail, Maildir uses individual files for each email. This lends itself to incremental updates, tools like <code class="highlighter-rouge">grep</code>, and compression.</p>
<p>Few clients operate on Maildir directly, unfortunately. Definitely none which I’m comfortable using (sorry, Mutt).</p>
<p>It’s quite easy to push Maildir back into any IMAP-supported host with <code class="highlighter-rouge">mbsync</code> should you need to restore from a backup. And if you really need an always-on, self-hosted client you can push into one as part of your sync.</p>
<h4 id="docker">Docker</h4>
<p>In order to automate this procedure I wrapped <code class="highlighter-rouge">mbsync</code> up in a Docker container as <a href="https://github.com/JakeWharton/docker-mbsync/">jakewharton/mbsync</a> which can run it on a periodic schedule.</p>
<p>It uses same <a href="https://healthchecks.io/">healthchecks.io</a> service as the <a href="https://github.com/bcardiff/docker-rclone">rclone</a> and <a href="https://github.com/JakeWharton/docker-gphotos-sync">gphotos-sync</a> containers from the last post for monitoring. I personally send this to my own Slack workspace which gives me simple history and easy notifications on all my devices.</p>
<p>Here’s its entry in my <code class="highlighter-rouge">docker-compose.yml</code>:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">version</span><span class="pi">:</span> <span class="s2">"</span><span class="s">3.6"</span>
<span class="na">services</span><span class="pi">:</span>
<span class="c1"># Services from previous blog post...</span>
<span class="na">mbsync-jake</span><span class="pi">:</span>
<span class="na">container_name</span><span class="pi">:</span> <span class="s">mbsync-jake</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">jakewharton/mbsync:latest</span>
<span class="na">restart</span><span class="pi">:</span> <span class="s">unless-stopped</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">/tanker/backup/jake/mail:/mail</span>
<span class="pi">-</span> <span class="s">${USERDIR}/docker/mbsync-jake:/config</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="c1"># Hourly</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">CRON=0</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">*"</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">CHECK_URL=https://hc-ping.com/..."</span>
</code></pre></div></div>
<p>For information on how to set up the container please see <a href="https://github.com/JakeWharton/docker-mbsync/#readme">the repo’s README</a>.</p>
<h4 id="storage">Storage</h4>
<p>Just like the <a href="/removing-google-as-a-single-point-of-failure/#data-storage">“Data Storage” and “Data Replication”</a> sections from the last post, the backup goes to a dedicated ZFS filesystem. This filesystem is regularly snapshotted to provide local history. The data and all its snapshots are also synchronized to <a href="https://rsync.net">rsync.net</a> for an off-site copy.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
tanker 18.4T 2.08T 151K /tanker
tanker/backup 529G 2.08T 151K /tanker/backup
tanker/backup/angela 172G 2.08T 140K /tanker/backup/angela
tanker/backup/angela/photos 172G 2.08T 172G /tanker/backup/angela/photos
tanker/backup/jake 337G 2.08T 151K /tanker/backup/jake
tanker/backup/jake/drive 78.9G 2.08T 78.9G /tanker/backup/jake/drive
tanker/backup/jake/mail 12.1G 2.08T 12.1G /tanker/backup/jake/mail
tanker/backup/jake/photos 246G 2.08T 148G /tanker/backup/jake/photos
</code></pre></div></div>
<p>I didn’t bother enabling compression on the filesystem because it’s only 12GiB. I suspect it would compress very well and it’s something that I can always turn on later.</p>
<hr />
<p>I did this migration one week after the previous post so I’ve been on Fastmail for about three weeks now. In general it’s been a positive experience. The Android app is hybrid so sometimes it feels a bit weird, but otherwise the clients have some nice features. My favorite so far is how it deals with quoted sections in long threads:</p>
<p><img src="/static/post-image/fastmail-quote.png" alt="Screenshot of Fastmail showing a large quoted section collapsed" style="border: 1px solid #ddd;" /></p>
<p>Having much more control over my email, photos, and files is comforting but I sincerely hope I never need to rely on these backups.</p>
<p>Once configured the Docker containers have been almost entirely maintenance free. I haven’t touched the photos or files sync for over a month now. Sometimes it hiccups and notifies me, but it’s always recovered on its own.</p>
<p><img src="/static/post-image/healthchecks.png" alt="Screenshot of Slack channel showing healthchecks.io notifications of sync being down and then recovering an hour later" style="border: 1px solid #222;" /></p>
<p>Now that Google is mostly removed as a single point of failure (I’m still relying on them for Keep and employment for now), it seems like getting automated backups rolling for all my GitHub projects is the next most pressing matter.</p>
Removing Google as a Single Point of Failure2020-02-19T00:00:00+00:00https://jakewharton.com/removing-google-as-a-single-point-of-failure<p>I want to remove Google as a single point of failure in my life. They have two decades of my email. They have two decades of my photos. They have the only copy of thousands of documents, projects, and other random files from the last two decades.</p>
<p>Now I trust Google <em>completely</em> in their ability to correctly retain my data. But I think it’s clear that over the last 5 years the company has lost something intrinsically important in the way it operates. I no longer trust them not to permanently lock me out of my account. And I say this as a current Google employee.</p>
<p>This year I’ve embarked on a mission to reclaim ownership of my data. This does not mean that I’m going to stop using Google products. Quite the opposite. Gmail, Google Photos, and Google Drive will remain the source-of-truth for all of the things I listed above. What’s different is that should Google disappear tomorrow (or just my account) I would lose no data.</p>
<h3 id="get-your-data">Get Your Data</h3>
<h4 id="step-1-takeout">Step 1: Takeout</h4>
<p>The first thing you need to do <strong>today</strong> is visit <a href="https://takeout.google.com/">takeout.google.com</a> and export your Gmail, Photos, and Drive data (and anything else you want). This will send you links to a set of 50GB <code class="highlighter-rouge">.tar.gz</code> files of your data that you can download.</p>
<p>That is, provided it works. It took me 5 attempts of exporting just my Photos data to have one succeed. Persistence pays off, though, so don’t give up even though this is a slow process. Get. Your. Data.</p>
<p>Google providing the Takeout service is amazing, but as far as a backup solutions go it is woefully inadequate. It’s an extremely manual, slow, and non-incremental process. However, it’s also comprehensive in ways that no other solution can match. Because of that, I have a monthly recurring task to perform a Takeout. Do it during a boring meeting so it feels less of a chore and more of a welcome distraction.</p>
<p>Seriously, do this today!</p>
<h4 id="step-2-drive-sync">Step 2: Drive Sync</h4>
<p>The <a href="https://rclone.org/">rclone</a> tool can incrementally sync your Google Drive contents. It will also take Google’s proprietary document formats and convert them into well-defined standard formats (which usually means Microsoft Office formats).</p>
<p>I run <code class="highlighter-rouge">rclone</code> hourly using the <a href="https://github.com/bcardiff/docker-rclone">bcardiff/docker-rclone</a> Docker container onto a large, redundant storage array (more on this array later). This container is nice because it pings <a href="https://healthchecks.io/">healthchecks.io</a> after each hourly sync. The check is set up to expect an hourly ping with a grace period of two hours.</p>
<h4 id="step-3-photo-sync">Step 3: Photo Sync</h4>
<p>While Google Photos does have an API, it <a href="https://issuetracker.google.com/issues/112096115">does not provide access to the original image</a>. This is some bullshit. I pay for Google Drive and Google Photos storage but I can only access original files for Drive. Some bullshit.</p>
<p>Thankfully, after <a href="https://twitter.com/JakeWharton/status/1222017202662125568">tweeting about said bullshit</a> I was pointed at the <a href="http://github.com/perkeep/gphotos-cdp">gphotos-cdp</a> tool (built by some <em>very</em> smart people). This uses the Chrome DevTools protocol to drive the Google Photos website and download the original photos one-by-one. This is awful and awesome and scary and it totally works!</p>
<p>In an effort to automate this, I wrapped the tool up in a Docker container as <a href="https://github.com/JakeWharton/docker-gphotos-sync">jakewharton/gphotos-sync</a> which can run it on a periodic schedule and uses the same <a href="https://healthchecks.io/">healthchecks.io</a> service as the rclone container. The initial setup is a little rough, but I’ve been running two instances hourly for two weeks without incident. Try it out!</p>
<h4 id="step-4-gmail-sync">Step 4: Gmail Sync</h4>
<p>I looked into a bunch of tools to do backup Gmail but I couldn’t find one that was still maintained and still worked. This is bad. Takeout is a start for this, but I want something more real-time.</p>
<p>Anyone have a solution here? Please let me know!</p>
<h3 id="data-storage">Data Storage</h3>
<p>I recently built a brand new home server with the intent of using it for storing backups of my Google data (among other things). It has four 8TB drives in a ZFS pool to ensure data is written to more than one physical drive. ZFS is an incredible storage technology that can ensure data is written in a way that is resilient to both drive failures and bitrot on both the writing and reading side.</p>
<p>ZFS supports creating separate filesystems as easily as you would normally create folders. Each filesystem can manage things like storage quotas and their own snapshots of content. Each of the Docker containers running rclone or gphotos-cdp writes into its own ZFS filesystem.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
tanker 17.8T 2.71T 151K /tanker
tanker/backup 417G 2.71T 151K /tanker/backup
tanker/backup/angela 170G 2.71T 140K /tanker/backup/angela
tanker/backup/angela/photos 170G 2.71T 170G /tanker/backup/angela/photos
tanker/backup/jake 227G 2.71T 151K /tanker/backup/jake
tanker/backup/jake/drive 78.9G 2.71T 78.9G /tanker/backup/jake/drive
tanker/backup/jake/photos 148G 2.71T 148G /tanker/backup/jake/photos
</code></pre></div></div>
<p>I currently use <a href="https://www.znapzend.org/">znapzhot</a> to recursively create automatic snapshots of all these filesytems under “tanker/backup”. My current policy is:</p>
<ul>
<li>Hourly snapshots retained for one day.</li>
<li>Daily snapshots retained for one month.</li>
<li>Monthly snapshots retained for one year.</li>
</ul>
<p>This policy is a hedge against any deleted or changed file. The simplicity of <code class="highlighter-rouge">cd</code>-ing into the hidden <code class="highlighter-rouge">.zfs</code> directory means these older copies are easily browsed, if ever needed.</p>
<h3 id="data-replication">Data Replication</h3>
<p>The frequently-repeated, best-practice rule for data storage is the “3–2–1 rule”. That is: three copies of the data, across two storage mediums, with one off-site location. In this framework, Google serves as one copy, one storage medium, and one off-site location. The local backups that we’re synchronizing serve as a second copy and a second storage medium (HDDs vs. the cloud).</p>
<p>For the third copy, I chose <a href="https://rsync.net/">rsync.net</a> which is quite the nerdy backup solution. Normally turning back to rclone for synchronizing the data to Dropbox or Backblaze would be an obvious solution. But rsync.net is unique in that they give you direct access to a ZFS zpool over SSH as root. This means that I can not only synchronize the latest data, but I can also synchronize the historical snapshots of it from the last year. The znapzend tool that I am already using handles sending the incremental snapshots as they’re taken. While rsync.net is a slightly more expensive alternative for cloud storage, the raw ZFS access and ability to store historical snapshots makes it worthwhile.</p>
<h3 id="self-hosting">Self Hosting</h3>
<p>In the unlikely event that Google implodes (or the far-more-likely scenario that they lock you out of your account) your data may be backed up but is otherwise relatively inaccessible. This is not very useful.</p>
<p>So far I have been serving read-only copies of my “tanker/backup” folder using NextCloud via the <a href="https://github.com/linuxserver/docker-nextcloud">linuxserver/nextcloud</a> Docker container. This not only affords me access on the go, but I can also easily share content with others.</p>
<p>NextCloud is a generic file host that offers document editing, photo viewing, and video playback in addition to just serving raw files. It offers many similar features to Google Drive. For example, if you do not want to set up the gphotos-cdp tool to back up your photos, you can run the NextCloud app on your phone which can automatically synchronize new photos to your server.</p>
<p>In order to expose NextCloud to the internet, you need, at minimum, knowledge of your IP address. While I do have business internet at home, I don’t have a static IP. Instead, I use the <a href="https://github.com/oznu/docker-cloudflare-ddns">oznu/cloudflare-ddns</a> Docker container to update a Cloudflare DNS A record on one of my domains.</p>
<p>Instead of exposing NextCloud directly to the internet, I use the <a href="https://containo.us/traefik/">traefik</a> Docker container as a reverse proxy. It takes care of talking to Let’s Encrypt to keep a valid SSL certificate in rotation as well as routing traffic for the domain to the NextCloud container.</p>
<h3 id="docker">Docker</h3>
<p>The NextCloud, Traefik, Cloudflare DDNS, rclone, and gphotos-cdp containers are all managed by Docker Compose. This makes it easy to update and manage their configuration.</p>
<p>In order to monitor the host I also run <a href="https://netdata.cloud/">Netdata</a> and <a href="https://www.portainer.io">Portainer</a>.</p>
<p>Here’s my <code class="highlighter-rouge">docker-compose.yml</code>:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">version</span><span class="pi">:</span> <span class="s2">"</span><span class="s">3.6"</span>
<span class="na">services</span><span class="pi">:</span>
<span class="na">portainer</span><span class="pi">:</span>
<span class="na">container_name</span><span class="pi">:</span> <span class="s">portainer</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">portainer/portainer</span>
<span class="na">command</span><span class="pi">:</span> <span class="s">-H unix:///var/run/docker.sock</span>
<span class="na">restart</span><span class="pi">:</span> <span class="s">always</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">11080:9000"</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">${USERDIR}/docker/portainer/data:/data</span>
<span class="pi">-</span> <span class="s">/var/run/docker.sock:/var/run/docker.sock</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">TZ=${TZ}</span>
<span class="na">netdata</span><span class="pi">:</span>
<span class="na">container_name</span><span class="pi">:</span> <span class="s">netdata</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">netdata/netdata</span>
<span class="na">restart</span><span class="pi">:</span> <span class="s">unless-stopped</span>
<span class="na">hostname</span><span class="pi">:</span> <span class="s">netdata</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">19999:19999</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">PGID=998</span> <span class="c1">#docker group</span>
<span class="na">cap_add</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">SYS_PTRACE</span>
<span class="na">security_opt</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">apparmor:unconfined</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">${USERDIR}/docker/netdata:/etc/netdata:ro</span>
<span class="c1"># For monitoring:</span>
<span class="pi">-</span> <span class="s">/etc/passwd:/host/etc/passwd:ro</span>
<span class="pi">-</span> <span class="s">/etc/group:/host/etc/group:ro</span>
<span class="pi">-</span> <span class="s">/etc/os-release:/etc/os-release:ro</span>
<span class="pi">-</span> <span class="s">/proc:/host/proc:ro</span>
<span class="pi">-</span> <span class="s">/sys:/host/sys:ro</span>
<span class="pi">-</span> <span class="s">/var/log/smartd:/var/log/smartd:ro</span>
<span class="pi">-</span> <span class="s">/var/run/docker.sock:/var/run/docker.sock:ro</span>
<span class="na">traefik</span><span class="pi">:</span>
<span class="na">container_name</span><span class="pi">:</span> <span class="s">traefik</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">traefik</span>
<span class="na">command</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">--api.insecure=true"</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">--providers.docker=true"</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">--providers.docker.exposedbydefault=false"</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">--entrypoints.http.address=:80"</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">--entrypoints.https.address=:443"</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">--certificatesresolvers.letsencrypttls.acme.tlschallenge=true"</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">--certificatesresolvers.letsencrypttls.acme.email=example@example.com"</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">--certificatesresolvers.letsencrypttls.acme.storage=/letsencrypt/acme.json"</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">80:80"</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">443:443"</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">8080:8080"</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">${USERDIR}/docker/traefik/letsencrypt:/letsencrypt"</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">/var/run/docker.sock:/var/run/docker.sock:ro"</span>
<span class="na">labels</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">traefik.enable=true"</span>
<span class="c1"># HTTP-to-HTTPS Redirect</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">traefik.http.routers.http-catchall.entrypoints=http"</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">traefik.http.routers.http-catchall.rule=HostRegexp(`{host:.+}`)"</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">traefik.http.routers.http-catchall.middlewares=redirect-to-https"</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"</span>
<span class="na">cloudflare-ddns</span><span class="pi">:</span>
<span class="na">container_name</span><span class="pi">:</span> <span class="s">cloudflare-ddns</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">oznu/cloudflare-ddns</span>
<span class="na">restart</span><span class="pi">:</span> <span class="s">unless-stopped</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">API_KEY=apikey</span>
<span class="pi">-</span> <span class="s">ZONE=example.com</span>
<span class="pi">-</span> <span class="s">SUBDOMAIN=*</span>
<span class="na">nextcloud</span><span class="pi">:</span>
<span class="na">container_name</span><span class="pi">:</span> <span class="s">nextcloud</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">linuxserver/nextcloud</span>
<span class="na">restart</span><span class="pi">:</span> <span class="s">unless-stopped</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">TZ=${TZ}</span>
<span class="pi">-</span> <span class="s">PUID=${PUID}</span>
<span class="pi">-</span> <span class="s">PGID=${PGID}</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">${USERDIR}/docker/nextcloud:/config</span>
<span class="pi">-</span> <span class="s">/tanker/nextcloud:/data</span>
<span class="pi">-</span> <span class="s">/tanker/backup:/backup:ro</span>
<span class="na">labels</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">traefik.enable=true"</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">traefik.http.routers.nextcloud.rule=Host(`files.example.com`)"</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">traefik.http.routers.nextcloud.entrypoints=https"</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">traefik.http.routers.nextcloud.tls.certresolver=letsencrypttls"</span>
<span class="na">rclone-drive-jake</span><span class="pi">:</span>
<span class="na">container_name</span><span class="pi">:</span> <span class="s">rclone-drive-jake</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">pfidr/rclone</span>
<span class="na">restart</span><span class="pi">:</span> <span class="s">unless-stopped</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">${USERDIR}/docker/rclone-drive-jake:/config</span>
<span class="pi">-</span> <span class="s">/tanker/backup/jake/drive:/gdrive</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">UID=${PUID}"</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">GID=${PGID}"</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">TZ=${TZ}"</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">SYNC_SRC=gdrive:"</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">SYNC_DEST=/gdrive"</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">CHECK_URL=https://hc-ping.com/..."</span>
<span class="c1"># Hourly</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">CRON=0</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">*"</span>
<span class="c1"># TODO update to https://github.com/rclone/rclone/issues/2893 when released</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">SYNC_OPTS=-v</span><span class="nv"> </span><span class="s">--drive-alternate-export"</span>
<span class="na">gphotos-sync-jake</span><span class="pi">:</span>
<span class="na">container_name</span><span class="pi">:</span> <span class="s">gphotos-sync-jake</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">jakewharton/gphotos-sync:latest</span>
<span class="na">restart</span><span class="pi">:</span> <span class="s">unless-stopped</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">${USERDIR}/docker/gphotos-sync-jake:/tmp/gphotos-cdp</span>
<span class="pi">-</span> <span class="s">/tanker/backup/jake/photos:/download</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">TZ=${TZ}</span>
<span class="c1"># Hourly</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">CRON=0</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">*"</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">CHECK_URL=https://hc-ping.com/..."</span>
<span class="na">gphotos-sync-angela</span><span class="pi">:</span>
<span class="na">container_name</span><span class="pi">:</span> <span class="s">gphotos-sync-angela</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">jakewharton/gphotos-sync:latest</span>
<span class="na">restart</span><span class="pi">:</span> <span class="s">unless-stopped</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">${USERDIR}/docker/gphotos-sync-angela:/tmp/gphotos-cdp</span>
<span class="pi">-</span> <span class="s">/tanker/backup/angela/photos:/download</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">TZ=${TZ}</span>
<span class="c1"># Hourly</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">CRON=0</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">*"</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">CHECK_URL=https://hc-ping.com/..."</span>
</code></pre></div></div>
<p>All of the containers store their configuration in <code class="highlighter-rouge">${USERDIR}/docker</code> which is in my home directory. This folder is mounted as a ZFS filesystem on a partition of the OS drive. It has a znapzend snapshot policy, is replicated into <code class="highlighter-rouge">/tanker/backup/home</code>, and is synchronized to rsync.net. In the event of this machine failing or being destroyed it should be fairly easy to set up a replacement.</p>
<hr />
<p>So far I’m pretty happy with this setup for backing up my Google Drive and Photos content. The apps for Drive and Photos are best-in-class and so I prefer to keep using them as the source of truth as long as possible. It’s nice to know that NextCloud could step in here if needed, but hopefully it never comes to that.</p>
<p>Gmail backups remain a problem to be solved. It’s also a <em>huge</em> problem that I cannot take control of my email address if it were needed. The Gmail webapp and mobile app also haven’t seen innovation in a decade and increasingly feel like legacy software. The thought of migrating my email is daunting, but it feels like it’s looming.</p>
<p>I continue to beleive that trusting Google with your data is a safe bet, but it is not a sufficient backup strategy by itself. Take control of your data.</p>
Extracting 100% of Data From a Stubborn, Dying ZFS Pool2020-02-12T00:00:00+00:00https://jakewharton.com/extracting-100-percent-of-data-from-a-stubborn-dying-zfs-zpool<p>In 2010 I built a home server with five 2TB drives. It ran Solaris and ZFS for the redundancy and data checksumming to ensure no data could be lost or corrupted. Just 16 months later five 3TB drives were added to the pool. This computer took the 2600-mile trip to live in San Francisco with me. It then endured the 2600-mile return trip when I left.</p>
<p>Having sat unplugged for five years, I recently powered the server back on for new workloads. But relying on 10 ten-year-old hard drives in 2020 is asking for cascading failure. And not only were the drives old, they’ve experienced physical trauma. So instead I built a new server and endeavored to migrate the data.</p>
<p>During the transfer the drives exhibited consistent read failures as expected, but ZFS was able to transparently mitigate them. Occasionally, though, the pool would lock up in a way that could only be fixed with a hard reboot. These lock ups sent me on a weird journey of software and hardware orchestration to complete the data transfer.</p>
<h3 id="symptoms">Symptoms</h3>
<p>During transfer of the data, progress would stall randomly in a way that seemingly could not be killed. CTRL+C had no effect. No <code class="highlighter-rouge">kill</code> signal had an effect. Even last-resort <code class="highlighter-rouge">shutdown -r now</code>s did nothing.</p>
<p>The system was oddly otherwise responsive. You could SSH in from another tab and poke around. <code class="highlighter-rouge">ps</code> showed that the transfer process was in the “D+” state which was uninterruptible sleep in the foreground.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>jake 21749 1.1 0.0 8400 2124 pts/0 D+ 23:42 0:00 rsync ...
</code></pre></div></div>
<p>That explained why the process wouldn’t die. The <code class="highlighter-rouge">dmesg</code> output also confirmed the problem happened deep in the I/O stack.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[ 3626.101527] INFO: task rsync:30680 blocked for more than 120 seconds.
[ 3626.101547] Tainted: P O 5.3.0-26-generic #28-Ubuntu
[ 3626.101563] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3626.101580] rsync D 0 30680 1 0x00000000
[ 3626.101584] Call Trace:
[ 3626.101590] __schedule+0x2b9/0x6c0
[ 3626.101596] schedule+0x42/0xb0
[ 3626.101601] schedule_timeout+0x152/0x2f0
[ 3626.101717] ? __next_timer_interrupt+0xe0/0xe0
[ 3626.101730] io_schedule_timeout+0x1e/0x50
[ 3626.101756] __cv_timedwait_common+0x15e/0x1c0 [spl]
[ 3626.101767] ? wait_woken+0x80/0x80
[ 3626.101790] __cv_timedwait_io+0x19/0x20 [spl]
[ 3626.102007] zio_wait+0x11b/0x230 [zfs]
[ 3626.102166] dmu_buf_hold_array_by_dnode+0x1db/0x490 [zfs]
[ 3626.102322] dmu_read_uio_dnode+0x49/0xf0 [zfs]
[ 3626.102523] ? zrl_add_impl+0x31/0xb0 [zfs]
[ 3626.102680] dmu_read_uio_dbuf+0x47/0x60 [zfs]
[ 3626.102880] zfs_read+0x117/0x300 [zfs]
[ 3626.103086] zpl_read_common_iovec+0x99/0xe0 [zfs]
[ 3626.103292] zpl_iter_read_common+0xa8/0x100 [zfs]
[ 3626.103496] zpl_iter_read+0x58/0x80 [zfs]
[ 3626.103509] new_sync_read+0x122/0x1b0
[ 3626.103525] __vfs_read+0x29/0x40
[ 3626.103536] vfs_read+0xab/0x160
[ 3626.103547] ksys_read+0x67/0xe0
[ 3626.103558] __x64_sys_read+0x1a/0x20
[ 3626.103569] do_syscall_64+0x5a/0x130
[ 3626.103581] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 3626.103594] RIP: 0033:0x7fe95027a272
[ 3626.103608] Code: Bad RIP value.
[ 3626.103616] RSP: 002b:00007ffd1cf2b8c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 3626.103629] RAX: ffffffffffffffda RBX: 00005649e1948980 RCX: 00007fe95027a272
[ 3626.103637] RDX: 0000000000040000 RSI: 00005649e1993120 RDI: 0000000000000004
[ 3626.103645] RBP: 0000000000040000 R08: 00000002a1df57af R09: 00000000108f57af
[ 3626.103654] R10: 00000000434ed0ba R11: 0000000000000246 R12: 0000000000000000
[ 3626.103662] R13: 0000000000040000 R14: 0000000000000000 R15: 0000000000000000
</code></pre></div></div>
<p>Visual inspection of the computer also showed anywhere from one to three drive LEDs were solid. None of the drive arms were moving to actually access data (the platters were still spinning).</p>
<p><img src="/static/post-image/transfer-leds.gif" height="512px" alt="Picture of stuck LEDs on hard drive cage" /></p>
<p>If there was a way to recover from this state I could not find it.</p>
<h3 id="mitigation">Mitigation</h3>
<p>The only way out of this state was a hard reboot which required holding down the power button. But after a reboot another lockup would occur randomly from within 10 seconds, to a few minutes in, to sometimes many hours later. Since I was only here to get data off of the machine, I did not want to spend a lot of time fixing the problem when a simple hard reboot was enough to unblock progress.</p>
<p>While seemingly random, I estimated a 100 hard reboots would be all that was needed. This meant progress could only be made during the day which increased transfer time from about 4 days to 10. Coupled with the restart time and then resuming transfer it looked more like 12 days would be required.</p>
<p>I decided this was unfortunate, but doable. I would work from my basement next to the machine which was hooked up to display on a TV. Whenever I noticed it was locked up, I would hard reboot the machine, wait for it to boot, and then resume the transfer by typing on its keyboard. In two weeks it would be over with.</p>
<p>After one day of working next to the machine I knew I needed to find a better solution. That day it had locked up 20-30 times which was triple what I had estimated for one day. Not only were the occurrences more frequent, but it took me a while to notice and using a keyboard attached to the server to restart the transfer was tedious.</p>
<p>In order for this transfer to complete with my sanity intact I needed to somehow automate the process. There were two problems to solve: figuring out when <code class="highlighter-rouge">rsync</code> was hung and performing a hard reboot.</p>
<h3 id="automating-detection">Automating Detection</h3>
<p>Because <code class="highlighter-rouge">rsync</code> was stuck deep in the I/O stack there was no chance for timeouts to apply. Neither its transfer-based timeout or a timeout on the SSH connection would cause it to exit when hung. Even when I moved <code class="highlighter-rouge">rsync</code> to run from the new machine I could not get a timeout to trigger. Running on the new machine did allow killing the process, which was a start.</p>
<p>Since <code class="highlighter-rouge">rsync</code> wouldn’t exit normally, I decided to automate hung detection the same way I checked manually: monitoring the output. If the last two lines of the output (which show the current file and transfer progress) haven’t changed in 10 seconds we consider the transfer to be hung.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Clear state from previous runs to ensure we can detect immediate hangs.</span>
<span class="nb">truncate</span> <span class="nt">-s</span> 0 rsync.txt
rsync <span class="nt">-ahHv</span> <span class="nt">--progress</span> <span class="se">\</span>
theflame:/tanker/<span class="k">*</span> <span class="se">\</span>
/tanker | <span class="nb">tee </span>rsync.txt &
<span class="nv">LAST_OUTPUT</span><span class="o">=</span><span class="s2">""</span>
<span class="k">while </span><span class="nb">sleep </span>10
<span class="k">do
</span><span class="nv">NEW_OUTPUT</span><span class="o">=</span><span class="si">$(</span><span class="nb">tail</span> <span class="nt">-2</span> rsync.txt<span class="si">)</span>
<span class="k">if</span> <span class="o">[[</span> <span class="s2">"</span><span class="nv">$LAST_OUTPUT</span><span class="s2">"</span> <span class="o">==</span> <span class="s2">"</span><span class="nv">$NEW_OUTPUT</span><span class="s2">"</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">break</span> <span class="c"># Output has not changed, assume locked up.</span>
<span class="k">fi
</span><span class="nv">LAST_OUTPUT</span><span class="o">=</span><span class="s2">"</span><span class="nv">$NEW_OUTPUT</span><span class="s2">"</span>
<span class="k">done</span>
</code></pre></div></div>
<p>This script will now exit when the transfer is hung. I could now detect hangs by playing beeps or sending myself a push notification with <code class="highlighter-rouge">curl</code> on exit. Using this script on the second day meant that almost no time was wasted waiting for me to notice the transfer had stopped. I was still hard rebooting the machine and re-starting the script 20-30 times, though.</p>
<h3 id="automating-reboot">Automating Reboot</h3>
<p>Last year I got a TP-Link Kasa Smart Plug as a stocking stuffer. I found a few sites which detailed how to use their undocumented API but while I was able to authenticate I was unable to toggle the power. Thankfully they have integration with IFTTT. I linked the plug and set up two applets which were each triggered by webhook.</p>
<p><img alt="IFTTT Applets for powering on and off the plug" src="/static/post-image/transfer-applets.png" height="141" /></p>
<p>I hooked the old server’s power through the plug and I could now control its power with two <code class="highlighter-rouge">curl</code> commands!</p>
<p><img alt="IFTTT Applets for powering on and off the plug" src="/static/post-image/transfer-plug.jpg" height="300" /></p>
<p>Integrating this into the script was a bit more complicated than expected. I started with a simple infinite loop running the above sync and then doing a power cycle with a delay.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">while</span> :
<span class="k">do</span>
<span class="c"># truncate, rsync, while loop from above...</span>
<span class="nb">echo</span> <span class="s2">"POWER: off"</span>
curl <span class="nt">-X</span> POST <span class="nt">-s</span> https://maker.ifttt.com/trigger/power_off/with/key/... <span class="o">></span> /dev/null
<span class="nb">sleep </span>5
<span class="nb">echo</span> <span class="s2">"POWER: on"</span>
curl <span class="nt">-X</span> POST <span class="nt">-s</span> https://maker.ifttt.com/trigger/power_on/with/key/... <span class="o">></span> /dev/null
<span class="nb">echo</span> <span class="s2">"Waiting 50s for startup..."</span>
<span class="nb">sleep </span>50
<span class="k">done</span>
</code></pre></div></div>
<p>After a few successful reboots the script would endlessly power-cycle the server or it would simply remain off. This was because IFTTT has no guarantees on latency or order of events. Instead of using <code class="highlighter-rouge">sleep</code> to time things, I switched to monitoring the actual machine for its state through SSH.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">while</span> :
<span class="k">do</span>
<span class="c"># Loop until SSH appears to confirm power on.</span>
ssh <span class="nt">-q</span> <span class="nt">-o</span> <span class="nv">ConnectTimeout</span><span class="o">=</span>1 theflame <span class="nb">exit
</span><span class="k">if</span> <span class="o">[[</span> <span class="nv">$?</span> <span class="o">!=</span> 0 <span class="o">]]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="s2">"SSH is not available. waiting 5s..."</span>
<span class="nb">sleep </span>5
<span class="k">continue
fi</span>
<span class="c"># truncate, rsync, while loop from above...</span>
<span class="nb">echo</span> <span class="s2">"POWER: off"</span>
curl <span class="nt">-X</span> POST <span class="nt">-s</span> https://maker.ifttt.com/trigger/power_off/with/key/... <span class="o">></span> /dev/null
<span class="k">while</span> :
<span class="k">do
</span><span class="nb">sleep </span>5
ssh <span class="nt">-q</span> <span class="nt">-o</span> <span class="nv">ConnectTimeout</span><span class="o">=</span>1 theflame <span class="nb">exit
</span><span class="k">if</span> <span class="o">[[</span> <span class="nv">$?</span> <span class="o">!=</span> 0 <span class="o">]]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">break</span> <span class="c"># SSH is down!</span>
<span class="k">fi
</span><span class="nb">echo</span> <span class="s2">"POWER: SSH is still available. Waiting 5s..."</span>
<span class="k">done
</span><span class="nb">echo</span> <span class="s2">"POWER: on"</span>
curl <span class="nt">-X</span> POST <span class="nt">-s</span> https://maker.ifttt.com/trigger/power_on/with/key/... <span class="o">></span> /dev/null
<span class="nb">echo</span> <span class="s2">"Waiting 50s for startup..."</span>
<span class="nb">sleep </span>50
<span class="k">done</span>
</code></pre></div></div>
<p>Now when IFTTT was delayed in delivering the <code class="highlighter-rouge">power_off</code> event the script would wait to confirm the machine powered off. This would sometimes spike as high as 10 minutes. But whenever it eventually triggered, 5 seconds later the <code class="highlighter-rouge">power_on</code> event would be sent and the machine would start coming back up. After 50 seconds it confirms SSH availability before restarting the <code class="highlighter-rouge">rsync</code>.</p>
<p>Sadly I didn’t capture a video of this in action. I did capture some of the output to excitedly share with some friends as they watched me go down this rabbit hole though.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Other/backup/presentations/...
1.20G 79% 79.29MB/s 0:00:25
POWER: off
POWER: on
Waiting 50s for startup...
Starting rsync...
receiving incremental file list
Other/
Other/backup/presentations/...
1.45G 95% 81.64MB/s 0:00:05
</code></pre></div></div>
<p>I let this script run for two days and it managed to complete the transfer of all files. IFTTT reports that the <code class="highlighter-rouge">power_on</code> and <code class="highlighter-rouge">power_off</code> events were each triggered 151 times! A few hours of scripting and a $10 gift saved me from two weeks of doing this myself.</p>
<h3 id="final-thoughts">Final Thoughts</h3>
<p>Had I known the drives would be so much trouble I would have taken a totally different approach. A better option would have been to run <code class="highlighter-rouge">dd</code> on the drives to copy their raw content to 2TB and 3TB files which would do a single pass across the platters. I think this would have been less likely to cause a freeze than the random access that <code class="highlighter-rouge">rsync</code> was doing. Then I could mount these files as storage on the new machine, import them as a ZFS pool, and do a local <code class="highlighter-rouge">zfs send | zfs recv</code> to get the data out.</p>
<p>I chose to use <code class="highlighter-rouge">rsync</code> over <code class="highlighter-rouge">zfs send | zfs recv</code> because I was unable to get a snapshot to complete before locking up. Once the initial transfer completed, I did a second pass using this script where <code class="highlighter-rouge">rsync</code> did a checksum of the file content on both ends (normally it only compares size and date). This found a few inconsistencies and re-transfered about 100GB of data.</p>
<p>Here’s the full script in its entirety: <a href="https://gist.github.com/JakeWharton/968859c48fd1bd7e85a0f78a164253b9">gist.github.com/JakeWharton/968859c48fd1bd7e85a0f78a164253b9</a>. There are some additional features such as mounting the old ZFS pool as readonly on boot and using a third IFTTT trigger to update a push notification on my phone for the current item.</p>
<p>More on what this new machine is for in future posts.</p>
<p><img src="/static/post-image/transfer-new.jpg" alt="Picture of new server" /></p>
D8 Library Desugaring2019-12-18T00:00:00+00:00https://jakewharton.com/d8-library-desugaring<blockquote>
<p>Note: This post is part of a series on D8 and R8, Android’s new dexer and optimizer, respectively. For an intro to D8 read <a href="/androids-java-8-support/">“Android’s Java 8 support”</a>. For an intro to R8 read <a href="/r8-optimization-staticization/">“R8 Optimization: Staticization”</a>.</p>
</blockquote>
<p>So far in this series the coverage of D8 has been about desugaring of <a href="/androids-java-8-support/">Java 8 language features</a>, working around <a href="/avoiding-vendor-and-version-specific-vm-bugs/">vendor- and version-specific bugs</a> in the platform, and performing <a href="/d8-optimizations/">method-local optimization</a>. In this post we’ll cover an upcoming feature of D8 called “core library desugaring” which makes newer APIs available on older versions of Android.</p>
<p>Library desugaring of Java 8 APIs such as streams, optional, and the new time APIs was announced at the developer keynote of Google I/O 2019 and delivered at Android DevSummit 2019 with the first canary build of Android Studio 4.0. This will allow developers to use these features introduced in API 24 and 26 on every version their app targets. No more backport libraries and duplicated APIs!</p>
<p>This is also a boon to the Java library ecosystem. Many libraries have long-since moved on to Java 8 but are unable to use newer APIs in order to maintain Android compatibility. While every new API is not available, D8 desugaring should allow these libraries to use the APIs which are most desired.</p>
<h3 id="not-a-new-feature">Not a new feature</h3>
<p>Despite the recent fanfare, desugaring APIs is not a actually a new feature of D8. Since it became a usable alternative to <code class="highlighter-rouge">dx</code>, D8 has desugared calls to the API 19 <code class="highlighter-rouge">Objects.requireNonNull</code> method. But, why that one method?</p>
<p>Certain code patterns will cause the Java compiler to synthesize an explicit null check.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Counter</span> <span class="o">{</span>
<span class="kd">final</span> <span class="kt">int</span> <span class="n">count</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">class</span> <span class="nc">Main</span> <span class="o">{</span>
<span class="kt">void</span> <span class="nf">doSomething</span><span class="o">(</span><span class="nc">Counter</span> <span class="n">counter</span><span class="o">)</span> <span class="o">{</span>
<span class="kt">int</span> <span class="n">count</span> <span class="o">=</span> <span class="n">counter</span><span class="o">.</span><span class="na">count</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>When compiled with JDK 8, the Java bytecode of the <code class="highlighter-rouge">doSomething</code> method contains a call to <code class="highlighter-rouge">getClass()</code> whose return value is then thrown away.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>void doSomething(Counter);
Code:
0: aload_1
1: invokevirtual #2 // Method java/lang/Object.getClass:()Ljava/lang/Class;
4: pop
5: iconst_0
6: istore_2
⋮
</code></pre></div></div>
<p>The zero value of <code class="highlighter-rouge">count</code> gets inlined into <code class="highlighter-rouge">doSomething</code> at bytecode index 5. As a result, if you were to pass <code class="highlighter-rouge">null</code> as the <code class="highlighter-rouge">Counter</code> the program would not throw a null-pointer exception. By including a call to <code class="highlighter-rouge">getClass()</code> on the <code class="highlighter-rouge">Counter</code>, the correct program behavior is maintained.</p>
<p>If you recompile this snippet with JDK 9, the bytecode changes.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> void doSomething(Counter);
Code:
0: aload_1
<span class="gd">- 1: invokevirtual #2 // Method java/lang/Object.getClass:()Ljava/lang/Class;
</span><span class="gi">+ 1: invokestatic #2 // Method java/util/Objects.requireNonNull:(Ljava/lang/Object;)Ljava/lang/Object;
</span> 4: pop
5: iconst_0
6: istore_2
⋮
</code></pre></div></div>
<p><a href="https://bugs.openjdk.java.net/browse/JDK-8074306">JDK-8074306</a> changed the behavior of the Java compiler in this scenario to produce better exceptions. But the Android toolchain has historically not worked correctly with JDK 9 (and newer), so you may be wondering how these calls came to be.</p>
<p>The primary source was <a href="https://errorprone.info/">Google’s error-prone</a> compiler and static analyzer which works with JDK 8 but is built on top of the JDK 9 compiler. While error-prone resolved <a href="https://github.com/google/error-prone/issues/375">the issue</a> by introducing an off-by-default flag, Retrolambda <a href="https://github.com/luontola/retrolambda/issues/75">added desugaring</a> for the API which basically required that D8 do the same.</p>
<p>Running D8 on the Java bytecode (with a minimum API level of less than 19) desugars the call back into a <code class="highlighter-rouge">getClass()</code> invocation.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[00016c] Main.doSomething:(LCounter;)V
0000: invoke-virtual {v1}, Ljava/lang/Object;.getClass:()Ljava/lang/Class;
⋮
</code></pre></div></div>
<p><code class="highlighter-rouge">Objects.requireNonNull</code> was the only API that D8 was able to desugar for a long time, and it did so using a simple rewrite. But soon its desugaring capabilities would have to expand in order to actually backport functionality.</p>
<h3 id="kotlins-java-8">Kotlin’s Java 8</h3>
<p>Unlike the Java compiler, the Kotlin compiler emits references to many APIs when generating bytecode for its language features. A <code class="highlighter-rouge">data class</code> is an example of the compiler generating a lot of bytecode on your behalf.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">data class</span> <span class="nc">Duration</span><span class="p">(</span><span class="kd">val</span> <span class="py">amount</span><span class="p">:</span> <span class="nc">Long</span><span class="p">,</span> <span class="kd">val</span> <span class="py">unit</span><span class="p">:</span> <span class="nc">TimeUnit</span><span class="p">)</span>
</code></pre></div></div>
<p>In Kotlin 1.1.60, when targeting Java 8 bytecode, the <code class="highlighter-rouge">hashCode</code> method of a <code class="highlighter-rouge">data class</code> changed to start referencing some Java 8 APIs.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>public int hashCode();
Code:
0: aload_0
1: getfield #10 // Field amount:J
4: invokestatic #71 // Method java/lang/Long.hashCode:(J)I
⋮
</code></pre></div></div>
<p>The compiler is free to call <code class="highlighter-rouge">Long.hashCode</code> because we told it that we were targeting Java 8. This is a new static method which has been added to the <code class="highlighter-rouge">Long</code> class.</p>
<p>Normally this would not be a problem for Android since the Kotlin compiler targets Java 6 by default. Unfortunately, the community push to target Java 8 for its language features interacted poorly with a decision to have the Kotlin compiler respect the specified target of your Java compiler in Kotlin 1.3. As a result, Android developers started seeing <code class="highlighter-rouge">NoSuchMethodError</code>s for these <code class="highlighter-rouge">hashCode</code> calls because they were only available in API 24 and newer.</p>
<p>While the behavior of the Kotlin compiler was reverted for Android projects, there still was a potential for libraries consumed by Android projects to be targeting Java 8 and to reference these methods. The D8 team decided to step in and mitigate this problem by desugaring the <code class="highlighter-rouge">hashCode</code> APIs.</p>
<p>Running D8 on the Java bytecode (with a minimum API level of less than 24) shows the desugaring.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[0003e4] Duration.hashCode:()I
0000: iget-wide v0, v2, LDuration;.amount:J
0002: invoke-static {v0, v1}, L$r8$backportedMethods$utility$Long$1$hashCode;.hashCode:(J)I
⋮
</code></pre></div></div>
<p>I’m not sure how you expected <code class="highlighter-rouge">Long.hashCode</code> to be desugared, but I’m guessing it wasn’t to a class named <code class="highlighter-rouge">$r8$backportedMethods$utility$Long$1$hashCode</code>! Unlike <code class="highlighter-rouge">Objects.requireNonNull</code> which was rewritten to <code class="highlighter-rouge">getClass()</code> to produce the same observable behavior, <code class="highlighter-rouge">Long.hashCode</code> has an implementation which cannot be replicated with a trivial rewrite.</p>
<h3 id="backporting-methods">Backporting methods</h3>
<p>Inside of the D8 project, there are template implementations of each API that it can backport.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">final</span> <span class="kd">class</span> <span class="nc">LongMethods</span> <span class="o">{</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">int</span> <span class="nf">hashCode</span><span class="o">(</span><span class="kt">long</span> <span class="n">l</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="o">(</span><span class="kt">int</span><span class="o">)</span> <span class="o">(</span><span class="n">l</span> <span class="o">^</span> <span class="o">(</span><span class="n">l</span> <span class="o">>>></span> <span class="mi">32</span><span class="o">));</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The code for these APIs are either written from the Javadoc specification of the method or adapted from libraries like <a href="https://github.com/google/guava">Google Guava</a>. When D8 is built, these templates are automatically converted into abstract representations of the method body.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">static</span> <span class="nc">CfCode</span> <span class="nf">LongMethods_hashCode</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="nf">CfCode</span><span class="o">(</span>
<span class="cm">/* maxStack = */</span> <span class="mi">5</span><span class="o">,</span>
<span class="cm">/* maxLocals = */</span> <span class="mi">2</span><span class="o">,</span>
<span class="nc">ImmutableList</span><span class="o">.</span><span class="na">of</span><span class="o">(</span>
<span class="k">new</span> <span class="nf">CfLoad</span><span class="o">(</span><span class="nc">ValueType</span><span class="o">.</span><span class="na">LONG</span><span class="o">,</span> <span class="mi">0</span><span class="o">),</span>
<span class="k">new</span> <span class="nf">CfLoad</span><span class="o">(</span><span class="nc">ValueType</span><span class="o">.</span><span class="na">LONG</span><span class="o">,</span> <span class="mi">0</span><span class="o">),</span>
<span class="k">new</span> <span class="nf">CfConstNumber</span><span class="o">(</span><span class="mi">32</span><span class="o">,</span> <span class="nc">ValueType</span><span class="o">.</span><span class="na">INT</span><span class="o">),</span>
<span class="k">new</span> <span class="nf">CfLogicalBinop</span><span class="o">(</span><span class="nc">CfLogicalBinop</span><span class="o">.</span><span class="na">Opcode</span><span class="o">.</span><span class="na">Ushr</span><span class="o">,</span> <span class="nc">NumericType</span><span class="o">.</span><span class="na">LONG</span><span class="o">),</span>
<span class="k">new</span> <span class="nf">CfLogicalBinop</span><span class="o">(</span><span class="nc">CfLogicalBinop</span><span class="o">.</span><span class="na">Opcode</span><span class="o">.</span><span class="na">Xor</span><span class="o">,</span> <span class="nc">NumericType</span><span class="o">.</span><span class="na">LONG</span><span class="o">),</span>
<span class="k">new</span> <span class="nf">CfNumberConversion</span><span class="o">(</span><span class="nc">NumericType</span><span class="o">.</span><span class="na">LONG</span><span class="o">,</span> <span class="nc">NumericType</span><span class="o">.</span><span class="na">INT</span><span class="o">),</span>
<span class="k">new</span> <span class="nf">CfReturn</span><span class="o">(</span><span class="nc">ValueType</span><span class="o">.</span><span class="na">INT</span><span class="o">)));</span>
<span class="o">}</span>
</code></pre></div></div>
<p>When D8 is compiling bytecode and first encounters a call to <code class="highlighter-rouge">Long.hashCode</code>, it generates a class on-the-fly with a <code class="highlighter-rouge">hashCode</code> method whose body created by calling that factory method. Each <code class="highlighter-rouge">Long.hashCode</code> call is then rewritten to point at this newly-generated class.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Class #0 -
Class descriptor : 'L$r8$backportedMethods$utility$Long$1$hashCode;'
Access flags : 0x1401 (PUBLIC ABSTRACT SYNTHETIC)
Superclass : 'Ljava/lang/Object;'
Direct methods -
#0
name : 'hashCode'
type : '(J)I'
access : 0x1009 (PUBLIC STATIC SYNTHETIC)
00044c: |[00044c] $r8$backportedMethods$utility$Long$1$hashCode.hashCode:(J)I
00045c: 1300 2000 |0000: const/16 v0, #int 32
000460: a500 0200 |0002: ushr-long v0, v2, v0
000464: c202 |0004: xor-long/2addr v2, v0
000466: 8423 |0005: long-to-int v3, v2
000468: 0f03 |0006: return v3
</code></pre></div></div>
<p>This process allows the Java 8-targeting <code class="highlighter-rouge">data class</code> work on versions of Android prior to API 24. If you look closely, you can probably map each Dalvik bytecode back to the abstract representation and then back to the template source code.</p>
<p>It may sound overkill to generate one class per method but this ensures that there is only one implementation of each API that requires backporting. When using R8, these synthesized classes also participate in optimizations such as method inlining and class merging which ultimately reduce their impact.</p>
<p>D8 can desugar 98 individual APIs from Java 7 and Java 8 which were added to existing types. But why stop there?</p>
<p>Because of how easy it is to add these templates, D8 can also desugar an additional 58 individual APIs from Java 9, 10, and 11 on existing types. This potentially allows Java libraries to target even newer versions of Java and still be used on Android.</p>
<p>A full list of the APIs which are available to desugar can be found <a href="/static/files/d8_api_desugar_list.txt">here</a>. Most of these are already available in AGP 3.6.0.</p>
<h3 id="backporting-types">Backporting Types</h3>
<p>Types like <code class="highlighter-rouge">Optional</code>, <code class="highlighter-rouge">Function</code>, <code class="highlighter-rouge">Stream</code>, and <code class="highlighter-rouge">LocalDateTime</code> are just some of those added in Java 8 which came to Android in API 24 and API 26. Backporting these to work on older API levels is more complicated than what it took to backport a single method for a few reasons.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Main</span> <span class="o">{</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">...</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="nc">LocalDateTime</span><span class="o">.</span><span class="na">now</span><span class="o">());</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p><code class="highlighter-rouge">LocalDateTime</code> was introduced in Android API 26 and an app whose minimum API level is 26 or higher can call into the class directly.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[000240] Main.main:([Ljava/lang/String;)V
0000: sget-object v1, Ljava/lang/System;.out:Ljava/io/PrintStream;
0002: invoke-static {}, Ljava/time/LocalDateTime;.now:()Ljava/time/LocalDateTime;
0005: move-result-object v0
0006: invoke-virtual {v1, v0}, Ljava/io/PrintStream;.println:(Ljava/lang/Object;)V
0009: return-void
</code></pre></div></div>
<p>To enable the use of these types when the minimum API is below 26, the Android Gradle plugin (4.0 or newer) requires that you enable “core library desugaring” in its DSL.</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">android</span> <span class="o">{</span>
<span class="n">compileOptions</span> <span class="o">{</span>
<span class="n">coreLibraryDesugaringEnabled</span> <span class="kc">true</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Recompiling will change the bytecode to reference the backport types.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> [000240] Main.main:([Ljava/lang/String;)V
0000: sget-object v1, Ljava/lang/System;.out:Ljava/io/PrintStream;
<span class="gd">-0002: invoke-static {}, Ljava/time/LocalDateTime;.now:()Ljava/time/LocalDateTime;
</span><span class="gi">+0002: invoke-static {}, Lj$/time/LocalDateTime;.now:()Lj$/time/LocalDateTime;
</span> 0005: move-result-object v0
0006: invoke-virtual {v1, v0}, Ljava/io/PrintStream;.println:(Ljava/lang/Object;)V
0009: return-void
</code></pre></div></div>
<p>The call to <code class="highlighter-rouge">java.time.LocalDateTime</code> was simply rewritten to <code class="highlighter-rouge">j$.time.LocalDateTime</code>, but the rest of the APK has changed dramatically.</p>
<p>Using the <a href="https://github.com/JakeWharton/diffuse/">diffuse tool</a> we can get a high-level view of the changes.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ diffuse diff app-min-26.apk app-min-25.apk
OLD: app-min-26.apk (signature: V2)
NEW: app-min-25.apk (signature: V2)
│ compressed │ uncompressed
├───────┬──────────┬───────────┼─────────┬──────────┬─────────
APK │ old │ new │ diff │ old │ new │ diff
──────────┼───────┼──────────┼───────────┼─────────┼──────────┼─────────
dex │ 680 B │ 44 KiB │ +43.4 KiB │ 944 B │ 90.9 KiB │ +90 KiB
arsc │ 524 B │ 520 B │ -4 B │ 384 B │ 384 B │ 0 B
manifest │ 603 B │ 603 B │ 0 B │ 1.2 KiB │ 1.2 KiB │ 0 B
other │ 229 B │ 229 B │ 0 B │ 95 B │ 95 B │ 0 B
──────────┼───────┼──────────┼───────────┼─────────┼──────────┼─────────
total │ 2 KiB │ 45.4 KiB │ +43.4 KiB │ 2.6 KiB │ 92.6 KiB │ +90 KiB
│ raw │ unique
├─────┬──────┬──────┼─────┬─────┬────────────────
DEX │ old │ new │ diff │ old │ new │ diff
─────────┼─────┼──────┼──────┼─────┼─────┼────────────────
count │ 1 │ 2 │ +1 │ │ │
strings │ 16 │ 1005 │ +989 │ 16 │ 996 │ +980 (+983 -3)
types │ 7 │ 175 │ +168 │ 7 │ 170 │ +163 (+164 -1)
classes │ 1 │ 88 │ +87 │ 1 │ 88 │ +87 (+87 -0)
methods │ 5 │ 728 │ +723 │ 5 │ 727 │ +722 (+724 -2)
fields │ 1 │ 255 │ +254 │ 1 │ 255 │ +254 (+254 -0)
</code></pre></div></div>
<p>There’s two important things that this summary tells us:</p>
<ol>
<li>Our APK size grew by 43.4KB which is entirely attributed to dex files. Looking at the dex changes there are a bunch of new classes, methods, and fields.</li>
<li>The number of dex files increased from one to two despite the number of total methods being nowhere close to the limit. These were release builds so we should be getting the minimum number of dex files.</li>
</ol>
<p>Let’s break each of these down.</p>
<h4 id="apk-size-impact">APK size impact</h4>
<p>Historically, in order to use the <code class="highlighter-rouge">java.time</code> APIs in an app with a minimum supported API level below 26 you would need to use the <a href="https://github.com/ThreeTen/threetenbp/">ThreeTenBP</a> library (or <a href="https://github.com/JakeWharton/ThreeTenABP/">ThreeTenABP</a>). This is a standalone repackaging of the <code class="highlighter-rouge">java.time</code> APIs in the <code class="highlighter-rouge">org.threeten.bp</code> package which requires you to update all your imports.</p>
<p>D8 is basically performing that same operation but at the bytecode level. It rewrites your code from calling <code class="highlighter-rouge">java.time</code> to <code class="highlighter-rouge">j$.time</code> as seen in the bytecode diff above. To accompany that rewrite, an implementation needs to be bundled into the application. That is the cause of the large APK size change.</p>
<p>In this example the release APK is minified using R8 which also minifies the backport code. If minification is disabled, the increase in dex size jumps up to 180KB, 206 classes, 3272 methods, and 713 fields.</p>
<h4 id="second-dex">Second Dex</h4>
<p>A release build will cause D8 or R8 to produce the minimum number of dex files required, and that’s actually still the case here. D8 and R8 are responsible for producing the dex files for user code and your declared libraries. This means that only the <code class="highlighter-rouge">Main</code> type will be present in the first dex which we can confirm by dumping its members.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ unzip app-min-25.apk classes.dex && \
diffuse members --dex --declared classes.dex
com.example.Main <init>()
com.example.Main main(String[])
</code></pre></div></div>
<p>As D8 or R8 are compiling your code and performing rewrites to the <code class="highlighter-rouge">j$</code> packages, they record the types and APIs that are being rewritten. This produces a set of shrinker rules that are specific to the backported types. Currently (i.e., for AGP 4.0.0-alpha06) these rules are located at <code class="highlighter-rouge">build/intermediates/desugar_lib_project_keep_rules/release/out/4</code> and for this example contains only the <code class="highlighter-rouge">LocalDateTime.now()</code> reference.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-keep class j$.time.LocalDateTime {
j$.time.LocalDateTime now();
}
</code></pre></div></div>
<p>All of the available backported types have been pre-compiled from OpenJDK source to a dex file as part of Google’s <a href="https://github.com/google/desugar_jdk_libs">desugar_jdk_libs</a> project. That dex file is downloaded from Google’s maven repo and then fed into a tool called L8 along with those generated keep rules. L8 shrinks this dex file in isolation using the provided rules to produce the final, second dex file.</p>
<p>Dumping the L8-minified second dex file shows a set of types and APIs that have been entirely obfuscated except for the <code class="highlighter-rouge">LocalDateTime.now()</code> API that the application is referencing.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ unzip app-min-25.apk classes2.dex && \
diffuse members --dex classes2.dex | grep -C 6 'LocalDateTime.now'
j$.time.LocalDateTime c(s) → long
j$.time.LocalDateTime compareTo(Object) → int
j$.time.LocalDateTime d() → h
j$.time.LocalDateTime d(s) → x
j$.time.LocalDateTime equals(Object) → boolean
j$.time.LocalDateTime hashCode() → int
j$.time.LocalDateTime now() → LocalDateTime
j$.time.LocalDateTime toString() → String
j$.time.a <init>(k)
j$.time.a a() → k
j$.time.a a: k
j$.time.a b() → f
j$.time.a c() → long
</code></pre></div></div>
<p>L8 is purpose-built for processing this special dex file. Previously in this series, R8 <a href="/r8-optimization-staticization/">was introduced</a> as…</p>
<blockquote>
<p>…a version of D8 that also performs optimization. It’s not a separate tool or codebase, just the same tool operating in a more advanced mode.</p>
</blockquote>
<p>Well L8 is a version of R8 that optimizes the JDK desugar dex file. It’s not a separate tool or codebase, just the same tool operating in a more advanced mode.</p>
<p>It may not be clear why the explicit extra dex is needed rather than consuming the desugared JDK types like any other library and allowing them to be processed normally by R8. First of all, Google probably doesn’t want me talking about it which should itself be somewhat of an indication why the extra ceremony is needed. For more information you can consult the OpenJDK source code license, specifically the very end. Sorry if that’s not enough information, but I suspect that’s all I’m allowed to say.</p>
<p>By virtue of always requiring at least a second dex, you either need have a minimum supported API of 21 or use <a href="https://developer.android.com/studio/build/multidex#mdex-pre-l">legacy multidex</a>. Most applications should choose the former, or use this feature as yet-another justification to potentially increase your minimum to 21.</p>
<h4 id="backporting-methods-on-backported-types">Backporting methods on backported types</h4>
<p>In addition to backporting methods on the types that have been around since API 1 like <code class="highlighter-rouge">Long</code>, D8 and R8 will also backport newer methods on these backportable types like <code class="highlighter-rouge">Optional</code>. These use the same template mechanism as detailed earlier, but will only be available when your minimum API level is high enough to access the target type or you have core library desugaring enabled.</p>
<p>For <code class="highlighter-rouge">Stream</code> and the four different optional types, D8 and R8 will backport 18 methods from Java 9, 10, and 11. The full list of those APIs can be found <a href="/static/files/d8_api_desugar_list_on_desugared_types.txt">here</a>.</p>
<h3 id="developer-story">Developer Story</h3>
<p>As a developer wanting to write code using these APIs, how do you know which ones are available for backport? Currently there’s not a great way to know about them all.</p>
<p>To start with, once you enable <code class="highlighter-rouge">coreLibraryDesugaring</code> the IDE and Lint will start allowing you to use the new types and new APIs when supported. Running Lint on this example will produce no errors despite the minimum supported API being below 26 which <code class="highlighter-rouge">LocalDateTime</code> would otherwise require. When library desugaring is disabled, though, the <code class="highlighter-rouge">NewApi</code> check fails as it normally would.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Main.java:7: Error: Call requires API level 26 (current min is 25): java.time.LocalDateTime#now [NewApi]
System.out.println(LocalDateTime.now());
~~~
</code></pre></div></div>
<p>This ensures you don’t errantly use an unsupported type or API, but it does not help for discoverability.</p>
<p>For now the best list of backported types is in the <a href="https://developer.android.com/studio/releases/gradle-plugin#j8-library-desugaring">Android Studio 4.0 feature list</a> and the best list of backported APIs on existing types are the two lists in this post (<a href="/static/files/d8_api_desugar_list.txt">1</a>, <a href="/static/files/d8_api_desugar_list_on_desugared_types.txt">2</a>). Hopefully in the future these will be more discoverable, though.</p>
<hr />
<p>The backporting of individual APIs has been improving since D8 and R8’s inception. With core library desugaring now becoming available in Android Gradle plugin 4.0 alphas, applications have access to the foundational types from Java 8 even when their minimum supported API level is lower than when those types were introduced. It also means that Java libraries can start to leverage these types while still maintaining compatibility with Android.</p>
<p>It’s important to remember that even with all this shiny new API availability, the JDK and Java APIs are continuing to improve along their six-month release cadence. While D8 and R8 can help bridge the gap by desugaring some of those APIs from Java 9, 10, and 11 even before they land in Android, pressure must be maintained to actually ship these APIs in the Android framework.</p>
Public API challenges in Kotlin2019-11-21T00:00:00+00:00https://jakewharton.com/public-api-challenges-in-kotlin<p>Kotlin is justifiably lauded for its language features compared to today’s Java. It has constructs which allow expressing common patterns with more concise alternatives. An overused example in every intro-to-Kotlin talk or blog post is comparing a Java “POJO” to a Kotlin <code class="highlighter-rouge">data class</code>.</p>
<p>Here’s yet another one of those comparisons, but bear with me as it will be used to illustrate the points in this post.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">final</span> <span class="kd">class</span> <span class="nc">Person</span> <span class="o">{</span>
<span class="kd">private</span> <span class="kd">final</span> <span class="nd">@NonNull</span> <span class="nc">String</span> <span class="n">name</span><span class="o">;</span>
<span class="kd">private</span> <span class="kd">final</span> <span class="kt">int</span> <span class="n">age</span><span class="o">;</span>
<span class="kd">public</span> <span class="nf">Person</span><span class="o">(</span><span class="nd">@NonNull</span> <span class="nc">String</span> <span class="n">name</span><span class="o">,</span> <span class="kt">int</span> <span class="n">age</span><span class="o">)</span> <span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">name</span> <span class="o">=</span> <span class="n">name</span><span class="o">;</span>
<span class="k">this</span><span class="o">.</span><span class="na">age</span> <span class="o">=</span> <span class="n">age</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="nd">@NonNull</span> <span class="nc">String</span> <span class="nf">getName</span><span class="o">()</span> <span class="o">{</span> <span class="k">return</span> <span class="n">name</span><span class="o">;</span> <span class="o">}</span>
<span class="kd">public</span> <span class="kt">int</span> <span class="nf">getAge</span><span class="o">()</span> <span class="o">{</span> <span class="k">return</span> <span class="n">age</span><span class="o">;</span> <span class="o">}</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="nc">String</span> <span class="nf">toString</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="s">"Person(name="</span> <span class="o">+</span> <span class="n">name</span> <span class="o">+</span> <span class="s">", age="</span> <span class="o">+</span> <span class="n">age</span> <span class="o">+</span> <span class="sc">')'</span>
<span class="o">}</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">equals</span><span class="o">(</span><span class="nd">@Nullable</span> <span class="nc">Object</span> <span class="n">o</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span><span class="n">o</span> <span class="o">==</span> <span class="k">this</span><span class="o">)</span> <span class="k">return</span> <span class="kc">true</span><span class="o">;</span>
<span class="k">if</span> <span class="o">(!(</span><span class="n">o</span> <span class="k">instanceof</span> <span class="nc">Person</span><span class="o">))</span> <span class="k">return</span> <span class="kc">false</span><span class="o">;</span>
<span class="nc">Person</span> <span class="n">other</span> <span class="o">=</span> <span class="o">(</span><span class="nc">Person</span><span class="o">)</span> <span class="n">o</span><span class="o">;</span>
<span class="k">return</span> <span class="n">name</span><span class="o">.</span><span class="na">equals</span><span class="o">(</span><span class="n">other</span><span class="o">.</span><span class="na">name</span><span class="o">)</span>
<span class="o">&&</span> <span class="n">age</span> <span class="o">==</span> <span class="n">other</span><span class="o">.</span><span class="na">age</span>
<span class="o">}</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">int</span> <span class="nf">hashCode</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="nc">Objects</span><span class="o">.</span><span class="na">hash</span><span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">age</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">data class</span> <span class="nc">Person</span><span class="p">(</span>
<span class="kd">val</span> <span class="py">name</span><span class="p">:</span> <span class="nc">String</span><span class="p">,</span>
<span class="kd">val</span> <span class="py">age</span><span class="p">:</span> <span class="nc">Int</span>
<span class="p">)</span>
</code></pre></div></div>
<p>Let us assume that this <code class="highlighter-rouge">Person</code> type is exposed in a library. As a result, evolving its public API needs to be done in a way that’s source and binary-compatible with previous versions. This post will cover some of the challenges of porting a library containing types like <code class="highlighter-rouge">Person</code> from Java to Kotlin while maintaining the required flexibility and exposing the correct conventions to each language.</p>
<h3 id="binary-compatibility">Binary Compatibility</h3>
<p>What changes are necessary in order to add a new property, <code class="highlighter-rouge">nickname</code>, to <code class="highlighter-rouge">Person</code> in a binary-compatible way?</p>
<p>For the manually-written Java type we add a new field, getter, and constructor parameter. In order to maintain compatibility, we retain the old constructor signature for old callers.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> public final class Person {
private final @NonNull String name;
<span class="gi">+ private final @Nullable String nickname;
</span> private final int age;
<span class="gd">- public Person(@NonNull String name, int age) {
</span><span class="gi">+ public Person(@NonNull String name, @Nullable String nickname, int age) {
</span> this.name = name;
<span class="gi">+ this.nickname = nickname;
</span> this.age = age;
}
<span class="gi">+ public Person(@NonNull String name, int age) {
+ this(name, null, age);
+ }
+
</span> public @NonNull String getName() { return name; }
<span class="gi">+ public @Nullable String getNickname() { return nickname; }
</span> public int getAge() { return age; }
@Override public String toString() {
<span class="gd">- return "Person(name=" + name + ", age=" + age + ')'
</span><span class="gi">+ return "Person(name=" + name + ", nickname=" + nickname + ", age=" + age + ')'
</span> }
@Override public boolean equals(@Nullable Object o) {
if (o == this) return true;
if (!(o instanceof Person)) return false;
Person other = (Person) o;
return name.equals(other.name)
<span class="gi">+ && Objects.equals(nickname, other.nickname)
</span> && age == other.age
}
@Override public int hashCode() {
<span class="gd">- return Objects.hash(name, age);
</span><span class="gi">+ return Objects.hash(name, nickname, age);
</span> }
}
</code></pre></div></div>
<p>So tedious!</p>
<p>The Kotlin class only needs a new property and the secondary constructor for compatibility.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> data class Person(
val name: String,
<span class="gi">+ val nickname: String?,
</span> val age: Int
<span class="gd">-)
</span><span class="gi">+) {
+ constructor(name: String, age: Int) : this(name, null, age)
+}
</span></code></pre></div></div>
<p>Much nicer, right? Unfortunately we have created two backwards-incompatible changes in the Kotlin version despite our efforts.</p>
<h4 id="destructuring-functions">Destructuring Functions</h4>
<p>For each property defined in the primary constructor, a data class will generate a <code class="highlighter-rouge">componentN()</code> function to facilitate <a href="https://kotlinlang.org/docs/reference/multi-declarations.html">destructuring declarations</a>. We can see these by running <code class="highlighter-rouge">javap</code> on the original Kotlin version of <code class="highlighter-rouge">Person</code>:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ javap Person.class
Compiled from "Person.kt"
public final class Person {
public final java.lang.String getName();
public final int getAge();
public final java.lang.String component1();
public final int component2();
⋮
</code></pre></div></div>
<p>Adding the <code class="highlighter-rouge">nickname</code> property in the middle of the primary constructor causes these component methods to shift incompatibly.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> public final class Person {
public final java.lang.String getName();
<span class="gi">+ public final java.lang.String getNickname();
</span> public final int getAge();
public final java.lang.String component1();
<span class="gd">- public final int component2();
</span><span class="gi">+ public final java.lang.String component2();
+ public final int component3();
</span> ⋮
</code></pre></div></div>
<p>Consumers who are destructuring <code class="highlighter-rouge">Person</code> will receive a <code class="highlighter-rouge">NoSuchMethodError</code> at runtime unless they also recompile their code.</p>
<p>We can work around this by only adding new properties at the end of the primary constructor. This will ensure that existing component methods do not change their return type.</p>
<p>A nice property of being forced to only append properties is that we can rely on default values and the <code class="highlighter-rouge">@JvmOverloads</code> annotation to avoid having to manually write secondary constructors.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gd">-data class Person(
</span><span class="gi">+data class Person @JvmOverloads constructor(
</span> val name: String,
val age: Int,
<span class="gi">+ val nickname: String? = null
</span> )
</code></pre></div></div>
<p>The downside of this approach is that you can no longer control the order of properties.</p>
<h4 id="copy-functions">Copy Functions</h4>
<p>In addition to the component functions, two <code class="highlighter-rouge">copy</code> functions are also generated automatically.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ javap Person.class
Compiled from "Person.kt"
public final class Person {
⋮
public final Person copy(java.lang.String, int);
public static Person copy$default(Person, java.lang.String, int, int, java.lang.Object);
⋮
</code></pre></div></div>
<p>These support creating a new instance of a <code class="highlighter-rouge">Person</code> while also updating a subset of its properties (e.g., <code class="highlighter-rouge">alice.copy(age = 99)</code>).</p>
<p>Unfortunately, adding the <code class="highlighter-rouge">nickname</code> property changes the signature of both of these methods breaking compatibility.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">public final class Person {
</span> ⋮
<span class="gd">- public final Person copy(java.lang.String, int);
</span><span class="gi">+ public final Person copy(java.lang.String, java.lang.String, int);
</span><span class="gd">- public static Person copy$default(Person, java.lang.String, int, int, java.lang.Object);
</span><span class="gi">+ public static Person copy$default(Person, java.lang.String, java.lang.String, int, int, java.lang.Object);
</span> ⋮
</code></pre></div></div>
<p>Even if you are only appending properties to avoid breaking the component functions, these two signatures <strong>always</strong> change. The use of <code class="highlighter-rouge">@JvmOverloads</code> on the primary constructor does not propagate to the <code class="highlighter-rouge">copy</code> functions. Any consumers using <code class="highlighter-rouge">copy</code> will now receive a <code class="highlighter-rouge">NoSuchMethodError</code> at runtime.</p>
<h4 id="mitigation-no-data">Mitigation: No <code class="highlighter-rouge">data</code></h4>
<p>The only real way to avoid these binary-incompatibilities for public API is to avoid the <code class="highlighter-rouge">data</code> modifier from the start and implement <code class="highlighter-rouge">equals</code>, <code class="highlighter-rouge">hashCode</code>, and <code class="highlighter-rouge">toString</code> yourself. Adding <code class="highlighter-rouge">nickname</code> to a non-data class can be now done in a fully-compatible way.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Person</span><span class="p">(</span>
<span class="kd">val</span> <span class="py">name</span><span class="p">:</span> <span class="nc">String</span><span class="p">,</span>
<span class="kd">val</span> <span class="py">nickname</span><span class="p">:</span> <span class="nc">String</span><span class="p">?,</span>
<span class="kd">val</span> <span class="py">age</span><span class="p">:</span> <span class="nc">Int</span>
<span class="p">)</span> <span class="p">{</span>
<span class="c1">// ...</span>
<span class="k">constructor</span><span class="p">(</span><span class="n">name</span><span class="p">:</span> <span class="nc">String</span><span class="p">,</span> <span class="n">age</span><span class="p">:</span> <span class="nc">Int</span><span class="p">)</span> <span class="p">:</span> <span class="k">this</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="k">null</span><span class="p">,</span> <span class="n">age</span><span class="p">)</span>
<span class="k">override</span> <span class="k">fun</span> <span class="nf">toString</span><span class="p">()</span> <span class="p">=</span> <span class="s">"Person(name=$name, nickname=$nickname, age=$age)"</span>
<span class="k">override</span> <span class="k">fun</span> <span class="nf">equals</span><span class="p">(</span><span class="n">other</span><span class="p">:</span> <span class="nc">Any</span><span class="p">?)</span> <span class="p">=</span> <span class="n">other</span> <span class="k">is</span> <span class="nc">Person</span>
<span class="p">&&</span> <span class="n">name</span> <span class="p">==</span> <span class="n">other</span><span class="p">.</span><span class="n">name</span>
<span class="p">&&</span> <span class="n">nickname</span> <span class="p">==</span> <span class="n">other</span><span class="p">.</span><span class="n">nickname</span>
<span class="p">&&</span> <span class="n">age</span> <span class="p">==</span> <span class="n">other</span><span class="p">.</span><span class="n">age</span>
<span class="k">override</span> <span class="k">fun</span> <span class="nf">hashCode</span><span class="p">()</span> <span class="p">=</span> <span class="nc">Objects</span><span class="p">.</span><span class="nf">hash</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">nickname</span><span class="p">,</span> <span class="n">age</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>You can implement the <code class="highlighter-rouge">componentN()</code> functions yourself to support destructuring. If you plan to add properties in the middle of the list, however, it may not make sense for the type to support destructuring.</p>
<p>The <code class="highlighter-rouge">copy</code> method can also be written manually, but evolving it compatibly is tricky. The simplest way is to maintain all of the old versions of the function but mark them as <code class="highlighter-rouge">@Deprecated(level=HIDDEN)</code>. This will keep their methods in the bytecode for old callers, but prevent new users from calling anything but the latest version.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Person</span><span class="p">(</span>
<span class="kd">val</span> <span class="py">name</span><span class="p">:</span> <span class="nc">String</span><span class="p">,</span>
<span class="kd">val</span> <span class="py">nickname</span><span class="p">:</span> <span class="nc">String</span><span class="p">?,</span>
<span class="kd">val</span> <span class="py">age</span><span class="p">:</span> <span class="nc">Int</span>
<span class="p">)</span> <span class="p">{</span>
<span class="c1">// ...</span>
<span class="nd">@Deprecated</span><span class="p">(</span><span class="s">""</span><span class="p">,</span> <span class="n">level</span> <span class="p">=</span> <span class="nc">HIDDEN</span><span class="p">)</span> <span class="c1">// For binary compatibility.</span>
<span class="k">fun</span> <span class="nf">copy</span><span class="p">(</span><span class="n">name</span><span class="p">:</span> <span class="nc">String</span> <span class="p">=</span> <span class="k">this</span><span class="p">.</span><span class="n">name</span><span class="p">,</span> <span class="n">age</span><span class="p">:</span> <span class="nc">Int</span> <span class="p">=</span> <span class="k">this</span><span class="p">.</span><span class="n">age</span><span class="p">)</span> <span class="p">=</span>
<span class="nf">copy</span><span class="p">(</span><span class="n">name</span> <span class="p">=</span> <span class="n">name</span><span class="p">,</span> <span class="n">age</span> <span class="p">=</span> <span class="n">age</span><span class="p">)</span> <span class="c1">// Calls the function below.</span>
<span class="k">fun</span> <span class="nf">copy</span><span class="p">(</span><span class="n">name</span><span class="p">:</span> <span class="nc">String</span> <span class="p">=</span> <span class="k">this</span><span class="p">.</span><span class="n">name</span><span class="p">,</span> <span class="n">nickname</span><span class="p">:</span> <span class="nc">String</span><span class="p">?</span> <span class="p">=</span> <span class="k">this</span><span class="p">.</span><span class="n">nickname</span><span class="p">,</span> <span class="n">age</span><span class="p">:</span> <span class="nc">Int</span> <span class="p">=</span> <span class="k">this</span><span class="p">.</span><span class="n">age</span><span class="p">)</span> <span class="p">=</span>
<span class="nc">Person</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">nickname</span><span class="p">,</span> <span class="n">age</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<h3 id="interop-compatibility">Interop Compatibility</h3>
<p>Another part of compatibility when migrating the <code class="highlighter-rouge">Person</code> library from Java to Kotlin is maintaining correct conventions for the API exposed to each language.</p>
<p>To avoid the explosion of constructors in Java, the <code class="highlighter-rouge">Person</code> type would traditionally hide its constructor and expose a nested <code class="highlighter-rouge">Builder</code> class. This not only allows adding new properties without a concern of binary compatibility, but allows properties to be supplied in any order and for partially-constructed instances to be passed around.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> public final class Person {
⋮
<span class="gd">- public Person(@NonNull String name, @Nullable String nickname, int age) {
</span><span class="gi">+ private Person(@NonNull String name, @Nullable String nickname, int age) {
</span> this.name = name;
this.nickname = nickname;
this.age = age;
}
⋮
<span class="gi">+
+ public static final class Builder {
+ private String name;
+ private String nickname;
+ private int age;
+
+ public Builder setName(String name) { this.name = name; }
+ public Builder setNickname(String nickname) { this.nickname = nickname; }
+ public Builder setAge(int age) { this.age = age; }
+
+ public Person build() {
+ return new Person(requireNonNull(name), nickname, age);
+ }
+ }
</span> }
</code></pre></div></div>
<p>Creating the builder in Kotlin is nearly identical.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gd">-class Person(
</span><span class="gi">+class Person private constructor(
</span> val name: String,
val nickname: String?,
val age: Int
) {
override fun toString() = TODO()
override fun equals(other: Any) = TODO()
override fun hashCode() = TODO()
<span class="gi">+
+ class Builder {
+ private var name: String? = null
+ private var nickname: String? = null
+ private var age: Int = 0
+
+ fun setName(name: String?) = apply { this.name = name }
+ fun setNickname(nickname: String?) = apply { this.nickname = nickname }
+ fun setAge(age: Int) = apply { this.age = age }
+
+ fun build() = Person(name!!, nickname, age)
+ }
</span> }
</code></pre></div></div>
<p>Nothing too interesting here, but by supporting Java we’re starting to create problems for Kotlin.</p>
<h4 id="builder-boilerplate">Builder Boilerplate</h4>
<p>A builder is usually a mutable(ish) version of an immutable type that also is responsible for validating any invariants (such as, in this case, that name is not null). It can be tempting to rewrite it in Kotlin as public <code class="highlighter-rouge">var</code>s to avoid the manual setter boilerplate.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> class Builder {
<span class="gd">- private var name: String? = null
</span><span class="gi">+ var name: String? = null
</span><span class="gd">- private var nickname: String? = null
</span><span class="gi">+ var nickname: String? = null
</span><span class="gd">- private var age: Int = 0
</span><span class="gi">+ var age: Int = 0
</span>
<span class="gd">- fun setName(name: String?) = apply { this.name = name }
- fun setNickname(nickname: String?) = apply { this.nickname = nickname }
- fun setAge(age: Int) = apply { this.age = age }
-
</span> fun build() = Person(name!!, nickname, age)
}
</code></pre></div></div>
<p>Unfortunately, doing so would be incorrect. The return type of the generated setters are now <code class="highlighter-rouge">void</code> instead of <code class="highlighter-rouge">Builder</code>.</p>
<p>Without a language change to allow property setters to return values, we are forced to use setter functions. I tend to keep the public <code class="highlighter-rouge">var</code> but hide its <code class="highlighter-rouge">void</code>-returning setter from Java with the <code class="highlighter-rouge">@JvmSynthetic</code> annotation. This allows Kotlin users to still get full usage of the property for reading and writing.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> class Builder {
<span class="gd">- private var name: String? = null
</span><span class="gi">+ @set:JvmSynthetic // Hide 'void' setter from Java
+ var name: String? = null
</span><span class="gd">- private var nickname: String? = null
</span><span class="gi">+ @set:JvmSynthetic // Hide 'void' setter from Java
+ var nickname: String? = null
</span><span class="gd">- private var age: Int = 0
</span><span class="gi">+ @set:JvmSynthetic // Hide 'void' setter from Java
+ var age: Int = 0
</span>
fun setName(name: String?) = apply { this.name = name }
fun setNickname(nickname: String?) = apply { this.nickname = nickname }
fun setAge(age: Int) = apply { this.age = age }
fun build() = Person(name!!, nickname, age)
}
</code></pre></div></div>
<p>There is no annotation to hide the setter functions from Kotlin callers. While not essential, they’re far better served by mutating the properties in an <code class="highlighter-rouge">apply { }</code> block.</p>
<h4 id="constructor">Constructor</h4>
<p>By virtue of making the primary constructor private we’ve removed the idiomatic means of creating a <code class="highlighter-rouge">Person</code> for Kotlin. Instead of a builder, Kotlin prefers default parameter values and named arguments. The <code class="highlighter-rouge">@JvmSynthetic</code> annotation can’t be used to hide constructors from Java, so we need to purse a different approach.</p>
<p>There is a convention of defining a top-level function whose name is the same as a type which we can use to replicate the constructor.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fun</span> <span class="nf">Person</span><span class="p">(</span><span class="n">name</span><span class="p">:</span> <span class="nc">String</span><span class="p">,</span> <span class="n">nickname</span><span class="p">:</span> <span class="nc">String</span><span class="p">?</span> <span class="p">=</span> <span class="k">null</span><span class="p">,</span> <span class="n">age</span><span class="p">:</span> <span class="nc">Int</span><span class="p">):</span> <span class="nc">Person</span> <span class="p">{</span>
<span class="k">return</span> <span class="nc">Person</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">nickname</span><span class="p">,</span> <span class="n">age</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Since this is a regular function and not a constructor, we can hide it from Java with <code class="highlighter-rouge">@JvmSynthetic</code>.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gi">+@JvmSynthetic // Hide from Java callers who should use Builder.
</span> fun Person(name: String, nickname: String? = null, age: Int): Person {
⋮
</code></pre></div></div>
<p>Once again, however, we’ve fallen into a binary compatibility trap. This signature has the same problem as the <code class="highlighter-rouge">copy</code> function that was generated for a data class.</p>
<p>Thankfully, since we wrote this function, the same mitigation trick can be used as outlined above for a manually-written <code class="highlighter-rouge">copy</code>. That is, we maintain the old versions of the function and mark them as <code class="highlighter-rouge">@Deprecated(level=HIDDEN)</code>.</p>
<p>These factory functions have no way of enforcing only named-parameter usage. As a result, they are vulnerable to source-incompatibility issues as arguments change position.</p>
<p>There’s also the problem of having to duplicate default values in each of these factory functions and the builder. A best practice would be to maintain defaults in private constants that could be re-used, but that requires additional discipline and continues to add boilerplate.</p>
<h4 id="mitigation-factory-dsl">Mitigation: Factory DSL?</h4>
<p>While currently unconventional, another potential workaround for the constructor problem is to change from a function-like syntax to a DSL-like syntax leveraging the <code class="highlighter-rouge">Builder</code>.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@JvmSynthetic</span> <span class="c1">// Hide from Java callers who should use Builder.</span>
<span class="k">fun</span> <span class="nf">Person</span><span class="p">(</span><span class="n">initializer</span><span class="p">:</span> <span class="nc">Person</span><span class="p">.</span><span class="nc">Builder</span><span class="p">.()</span> <span class="p">-></span> <span class="nc">Unit</span><span class="p">):</span> <span class="nc">Person</span> <span class="p">{</span>
<span class="k">return</span> <span class="nc">Person</span><span class="p">.</span><span class="nc">Builder</span><span class="p">().</span><span class="nf">apply</span><span class="p">(</span><span class="n">initializer</span><span class="p">).</span><span class="nf">build</span><span class="p">()</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Creation of an instance now looks more like inline-JSON.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">val</span> <span class="py">alice</span> <span class="p">=</span> <span class="nc">Person</span> <span class="p">{</span>
<span class="n">name</span> <span class="p">=</span> <span class="s">"Alice Alison"</span>
<span class="n">age</span> <span class="p">=</span> <span class="mi">99</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This also has the advantage of re-using any default values from the builder allowing them to be localized in one place.</p>
<p>DSLs tend to have specialized usage and are do not currently have widespread usage as factories. Their ability to enforce named usage and maintain source and binary compatibility as properties are introduced makes them an attractive solution, however.</p>
<h3 id="summary">Summary</h3>
<p>Using Kotlin types whose properties will change over time in public API requires extra care to maintain source and binary compatibility as well as an idiomatic API for each language.</p>
<ul>
<li>Avoid using the <code class="highlighter-rouge">data</code> modifier. Instead, implement <code class="highlighter-rouge">equals</code>, <code class="highlighter-rouge">hashCode</code>, and <code class="highlighter-rouge">toString</code> yourself for these value-based types.</li>
<li>Expose a builder for Java callers. Public <code class="highlighter-rouge">var</code>s are not enough, fluent setters need to be written.</li>
<li>Hide constructors and be mindful of factory function binary compatibility. Reusing the builders for a DSL-factory may be a way to avoid this.</li>
</ul>
<p>If your type is not going to change its properties over time (like a 2D point) you can ignore this advice and stick with a simple <code class="highlighter-rouge">data class</code>.</p>
<hr />
<p>Here is the final <code class="highlighter-rouge">Person</code> declaration for the public API of a library:</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Person</span> <span class="k">private</span> <span class="k">constructor</span><span class="p">(</span>
<span class="kd">val</span> <span class="py">name</span><span class="p">:</span> <span class="nc">String</span><span class="p">,</span>
<span class="kd">val</span> <span class="py">nickname</span><span class="p">:</span> <span class="nc">String</span><span class="p">?,</span>
<span class="kd">val</span> <span class="py">age</span><span class="p">:</span> <span class="nc">Int</span>
<span class="p">)</span> <span class="p">{</span>
<span class="k">override</span> <span class="k">fun</span> <span class="nf">toString</span><span class="p">()</span> <span class="p">=</span> <span class="s">"Person(name=$name, nickname=$nickname, age=$age)"</span>
<span class="k">override</span> <span class="k">fun</span> <span class="nf">equals</span><span class="p">(</span><span class="n">other</span><span class="p">:</span> <span class="nc">Any</span><span class="p">?)</span> <span class="p">=</span> <span class="n">other</span> <span class="k">is</span> <span class="nc">Person</span>
<span class="p">&&</span> <span class="n">name</span> <span class="p">==</span> <span class="n">other</span><span class="p">.</span><span class="n">name</span>
<span class="p">&&</span> <span class="n">nickname</span> <span class="p">==</span> <span class="n">other</span><span class="p">.</span><span class="n">nickname</span>
<span class="p">&&</span> <span class="n">age</span> <span class="p">==</span> <span class="n">other</span><span class="p">.</span><span class="n">age</span>
<span class="k">override</span> <span class="k">fun</span> <span class="nf">hashCode</span><span class="p">()</span> <span class="p">=</span> <span class="nc">Objects</span><span class="p">.</span><span class="nf">hash</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">nickname</span><span class="p">,</span> <span class="n">age</span><span class="p">)</span>
<span class="kd">class</span> <span class="nc">Builder</span> <span class="p">{</span>
<span class="err">@</span><span class="k">set</span><span class="p">:</span><span class="nc">JvmSynthetic</span> <span class="c1">// Hide 'void' setter from Java</span>
<span class="kd">var</span> <span class="py">name</span><span class="p">:</span> <span class="nc">String</span><span class="p">?</span> <span class="p">=</span> <span class="k">null</span>
<span class="err">@</span><span class="k">set</span><span class="p">:</span><span class="nc">JvmSynthetic</span> <span class="c1">// Hide 'void' setter from Java</span>
<span class="kd">var</span> <span class="py">nickname</span><span class="p">:</span> <span class="nc">String</span><span class="p">?</span> <span class="p">=</span> <span class="k">null</span>
<span class="err">@</span><span class="k">set</span><span class="p">:</span><span class="nc">JvmSynthetic</span> <span class="c1">// Hide 'void' setter from Java</span>
<span class="kd">var</span> <span class="py">age</span><span class="p">:</span> <span class="nc">Int</span> <span class="p">=</span> <span class="mi">0</span>
<span class="k">fun</span> <span class="nf">setName</span><span class="p">(</span><span class="n">name</span><span class="p">:</span> <span class="nc">String</span><span class="p">?)</span> <span class="p">=</span> <span class="nf">apply</span> <span class="p">{</span> <span class="k">this</span><span class="p">.</span><span class="n">name</span> <span class="p">=</span> <span class="n">name</span> <span class="p">}</span>
<span class="k">fun</span> <span class="nf">setNickname</span><span class="p">(</span><span class="n">nickname</span><span class="p">:</span> <span class="nc">String</span><span class="p">?)</span> <span class="p">=</span> <span class="nf">apply</span> <span class="p">{</span> <span class="k">this</span><span class="p">.</span><span class="n">nickname</span> <span class="p">=</span> <span class="n">nickname</span> <span class="p">}</span>
<span class="k">fun</span> <span class="nf">setAge</span><span class="p">(</span><span class="n">age</span><span class="p">:</span> <span class="nc">Int</span><span class="p">)</span> <span class="p">=</span> <span class="nf">apply</span> <span class="p">{</span> <span class="k">this</span><span class="p">.</span><span class="n">age</span> <span class="p">=</span> <span class="n">age</span> <span class="p">}</span>
<span class="k">fun</span> <span class="nf">build</span><span class="p">()</span> <span class="p">=</span> <span class="nc">Person</span><span class="p">(</span><span class="n">name</span><span class="o">!!</span><span class="p">,</span> <span class="n">nickname</span><span class="p">,</span> <span class="n">age</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nd">@JvmSynthetic</span> <span class="c1">// Hide from Java callers who should use Builder.</span>
<span class="k">fun</span> <span class="nf">Person</span><span class="p">(</span><span class="n">initializer</span><span class="p">:</span> <span class="nc">Person</span><span class="p">.</span><span class="nc">Builder</span><span class="p">.()</span> <span class="p">-></span> <span class="nc">Unit</span><span class="p">):</span> <span class="nc">Person</span> <span class="p">{</span>
<span class="k">return</span> <span class="nc">Person</span><span class="p">.</span><span class="nc">Builder</span><span class="p">().</span><span class="nf">apply</span><span class="p">(</span><span class="n">initializer</span><span class="p">).</span><span class="nf">build</span><span class="p">()</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Quite the distance from the simple <code class="highlighter-rouge">data class</code> version, but it’s at least safe to change over time.</p>
<p>Future versions of Kotlin will stabilize compiler plugins allowing these patterns to be placed behind annotations or custom modifiers.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Hypothetical 'value' on 'class' provides generated 'equals',</span>
<span class="c1">// 'hashCode', and 'toString' similar to 'data'.</span>
<span class="n">value</span> <span class="kd">class</span> <span class="nc">Person</span> <span class="k">private</span> <span class="k">constructor</span><span class="p">(</span>
<span class="kd">val</span> <span class="py">name</span><span class="p">:</span> <span class="nc">String</span><span class="p">,</span>
<span class="kd">val</span> <span class="py">nickname</span><span class="p">:</span> <span class="nc">String</span><span class="p">?</span> <span class="p">=</span> <span class="k">null</span><span class="p">,</span>
<span class="kd">val</span> <span class="py">age</span><span class="p">:</span> <span class="nc">Int</span>
<span class="p">)</span> <span class="p">{</span>
<span class="c1">// Hypothetical 'builder' on nested 'class' exposes mutable</span>
<span class="c1">// versions of primary constructor properties.</span>
<span class="n">builder</span> <span class="kd">class</span> <span class="nc">Builder</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This will eliminate the boilerplate required to create Kotlin types suitable for evolving in public APIs.</p>
D8 Optimizations2019-10-30T00:00:00+00:00https://jakewharton.com/d8-optimizations<blockquote>
<p>Note: This post is part of a series on D8 and R8, Android’s new dexer and optimizer, respectively. For an intro to D8 read <a href="/androids-java-8-support/">“Android’s Java 8 support”</a>. For an intro to R8 read <a href="/r8-optimization-staticization/">“R8 Optimization: Staticization”</a>.</p>
</blockquote>
<p>No, that’s not a typo! While the optimizations in this series so far have been done by R8 during whole-program optimization, D8 can also perform some simple optimizations.</p>
<p>D8 <a href="/androids-java-8-support/">was introduced</a> as the new Java-to-Dalvik bytecode compiler for Android. It handles backporting of Java 8 language features to work on Android (as well as <a href="/androids-java-9-10-11-and-12-support/">those of Java 9 and beyond</a>). It also works around <a href="/avoiding-vendor-and-version-specific-vm-bugs/">vendor- and version-specific bugs</a> in the platform.</p>
<p>That’s what we’ve seen from D8 so far in the series, but it has two other responsibilities that we’ll cover in this post and the next:</p>
<ol>
<li>Backporting methods to work on older API levels where they didn’t exist.</li>
<li>Performing local optimizations to reduce bytecode size and/or improve performance.</li>
</ol>
<p>We’ll cover API backporting in the next post in the series. For now, let’s look at some of the local optimizations that D8 might perform.</p>
<h3 id="switch-rewriting">Switch Rewriting</h3>
<p>The last two posts (<a href="/r8-optimization-enum-ordinals-and-names/">1</a>, <a href="/r8-optimization-enum-switch-maps/">2</a>) have dealt with optimizing switch statements. Both have slightly lied about the bytecode that D8 and R8 produce for certain switch statements. Let’s look at one of those examples again.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">enum</span> <span class="nc">Greeting</span> <span class="o">{</span>
<span class="no">FORMAL</span><span class="o">,</span> <span class="no">INFORMAL</span><span class="o">;</span>
<span class="kd">static</span> <span class="nc">String</span> <span class="nf">greetingType</span><span class="o">(</span><span class="nc">Greeting</span> <span class="n">greeting</span><span class="o">)</span> <span class="o">{</span>
<span class="k">switch</span> <span class="o">(</span><span class="n">greeting</span><span class="o">)</span> <span class="o">{</span>
<span class="k">case</span> <span class="nl">FORMAL:</span> <span class="k">return</span> <span class="s">"formal"</span><span class="o">;</span>
<span class="k">case</span> <span class="nl">INFORMAL:</span> <span class="k">return</span> <span class="s">"informal"</span><span class="o">;</span>
<span class="k">default</span><span class="o">:</span> <span class="k">throw</span> <span class="k">new</span> <span class="nc">AssertionError</span><span class="o">();</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The full Java bytecode that was shown for <code class="highlighter-rouge">greetingType</code> used the <code class="highlighter-rouge">lookupswitch</code> bytecode which has offsets for where to jump when a value is matched.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>static java.lang.String greetingType(Greeting);
Code:
0: getstatic #2 // Field Main$1.$SwitchMap$Greeting:[I
3: aload_0
4: invokevirtual #3 // Method Greeting.ordinal:()I
7: iaload
8: lookupswitch {
1: 36
2: 39
default: 42
}
36: ldc #4 // String formal
38: areturn
39: ldc #5 // String informal
41: areturn
42: new #6 // class java/lang/AssertionError
45: dup
46: invokespecial #7 // Method java/lang/AssertionError."<init>":()V
49: athrow
</code></pre></div></div>
<p>The <code class="highlighter-rouge">tableswitch</code> Java bytecode was shown as being rewritten to <code class="highlighter-rouge">packed-switch</code> when converted to Dalvik bytecode.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[000584] Main.greetingType:(LGreeting;)Ljava/lang/String;
0000: sget-object v0, LMain$1;.$SwitchMap$Greeting:[I
0002: invoke-virtual {v2}, LGreeting;.ordinal:()I
0005: move-result v1
0006: aget v0, v0, v1
0008: packed-switch v0, 00000017
000b: new-instance v0, Ljava/lang/AssertionError;
000d: invoke-direct {v0}, Ljava/lang/AssertionError;.<init>:()V
0010: throw v0
0011: const-string v0, "formal"
0013: return-object v0
0014: const-string v0, "informal"
0016: return-object v0
0017: packed-switch-data (8 units)
</code></pre></div></div>
<p>If we actually compile and dex the above source file with D8, its Dalvik bytecode output is different.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> [0005f0] Main.greetingType:(LGreeting;)Ljava/lang/String;
0000: sget-object v0, LMain$1;.$SwitchMap$Greeting:[I
0002: invoke-virtual {v1}, LGreeting;.ordinal:()I
0005: move-result v1
0006: aget v0, v0, v1
<span class="gd">-0008: packed-switch v0, 00000017
</span><span class="gi">+0008: const/4 v1, #int 1
+0009: if-eq v0, v1, 0014
+000b: const/4 v1, #int 2
+000c: if-eq v0, v1, 0017
</span> 000e: new-instance v0, Ljava/lang/AssertionError;
0010: invoke-direct {v0}, Ljava/lang/AssertionError;.<init>:()V
0013: throw v0
0014: const-string v0, "formal"
0016: return-object v0
0017: const-string v0, "informal"
0019: return-object v0
<span class="gd">-0017: packed-switch-data (8 units)
</span></code></pre></div></div>
<p>Instead of a <code class="highlighter-rouge">packed-switch</code> at bytecode index 0008, there are a series of <code class="highlighter-rouge">if</code>/<code class="highlighter-rouge">else if</code>-like checks. Based on the indices, you might think this winds up producing a larger binary but it’s actually the opposite. The original <code class="highlighter-rouge">packed-switch</code> is accompanied by a <code class="highlighter-rouge">packed-switch-data</code> bytecode that reports itself as being 8 units long. So the <code class="highlighter-rouge">packed-switch</code> version has a total cost of 26 bytecodes whereas the <code class="highlighter-rouge">if</code>/<code class="highlighter-rouge">else if</code> version only costs 20.</p>
<p>Rewriting switches to normal conditionals is only done when there is a bytecode savings. This depends on the number of <code class="highlighter-rouge">case</code> blocks, whether there’s fallthrough, and whether or not the values are contiguous or not. D8 computes the cost of both forms and then chooses that which is smaller.</p>
<h3 id="string-optimizations">String Optimizations</h3>
<p>Back in February there was a post on <a href="/r8-optimization-string-constant-operations/">R8’s string constants operations</a>. It showed an example from OkHttp where a call to <code class="highlighter-rouge">String.length</code> was made on a constant.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">static</span> <span class="nc">String</span> <span class="nf">patternHost</span><span class="o">(</span><span class="nc">String</span> <span class="n">pattern</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">pattern</span><span class="o">.</span><span class="na">startsWith</span><span class="o">(</span><span class="no">WILDCARD</span><span class="o">)</span>
<span class="o">?</span> <span class="n">pattern</span><span class="o">.</span><span class="na">substring</span><span class="o">(</span><span class="no">WILDCARD</span><span class="o">.</span><span class="na">length</span><span class="o">())</span>
<span class="o">:</span> <span class="n">pattern</span><span class="o">;</span>
<span class="o">}</span>
</code></pre></div></div>
<p>When compiled with the old <code class="highlighter-rouge">dx</code> tool the output is a straightforward translation.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[0001a8] Test.patternHost:(Ljava/lang/String;)Ljava/lang/String;
0000: const-string v0, "*."
0002: invoke-virtual {v2, v0}, Ljava/lang/String;.startsWith:(Ljava/lang/String;)Z
0005: move-result v1
0006: if-eqz v1, 0010
0008: invoke-virtual {v0}, Ljava/lang/String;.length:()I
0011: move-result v1
0012: invoke-virtual {v2, v1}, Ljava/lang/String;.substring:(I)Ljava/lang/String;
000f: move-result-object v2
0010: return-object v2
</code></pre></div></div>
<p>Bytecode index 0008 performs the <code class="highlighter-rouge">String.length</code> call on the constant loaded at index 0000.</p>
<p>With D8, however, this method call on a constant is detected and evaluated at compile-time to its corresponding numerical value.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> [0001a8] Test.patternHost:(Ljava/lang/String;)Ljava/lang/String;
0000: const-string v0, "*."
0002: invoke-virtual {v1, v0}, Ljava/lang/String;.startsWith:(Ljava/lang/String;)Z
0005: move-result v0
0006: if-eqz v0, 000d
<span class="gd">-0008: invoke-virtual {v0}, Ljava/lang/String;.length:()I
-0011: move-result v1
</span><span class="gi">+0008: const/4 v0, #int 2
</span> 0009: invoke-virtual {v1, v0}, Ljava/lang/String;.substring:(I)Ljava/lang/String;
000c: move-result-object v1
000d: return-object v1
</code></pre></div></div>
<p>Removing a method call is not something that D8 or even R8 will normally do. This optimization is only safe to apply because <code class="highlighter-rouge">String</code> is a final class in the framework with well-defined behavior.</p>
<p>In the nine months since the original post, the number of methods on a string which can be optimized has grown substantially. Both D8 and R8 will compute <code class="highlighter-rouge">isEmpty()</code>, <code class="highlighter-rouge">startsWith(String)</code>, <code class="highlighter-rouge">endsWith(String)</code>, <code class="highlighter-rouge">contains(String)</code>, <code class="highlighter-rouge">equals(String)</code>, <code class="highlighter-rouge">equalsIgnoreCase(String)</code>, <code class="highlighter-rouge">contentEquals(String)</code>, <code class="highlighter-rouge">hashCode()</code>, <code class="highlighter-rouge">length()</code>, <code class="highlighter-rouge">indexOf(String)</code>, <code class="highlighter-rouge">indexOf(int)</code>, <code class="highlighter-rouge">lastIndexOf(String)</code>, <code class="highlighter-rouge">lastIndexOf(int)</code>, <code class="highlighter-rouge">compareTo(String)</code>, <code class="highlighter-rouge">compareToIgnoreCase(String)</code>, <code class="highlighter-rouge">substring(int)</code>, <code class="highlighter-rouge">substring(int, int)</code>, and <code class="highlighter-rouge">trim()</code> on a constant string. Obviously it’s unlikely that most of these will apply without R8 inlining, but they’re there when it does occur.</p>
<h3 id="known-array-lengths">Known Array Lengths</h3>
<p>Just like how you might call <code class="highlighter-rouge">length()</code> on a constant string to maintain a single source of truth, it’s not uncommon to see code call <code class="highlighter-rouge">length</code> on an array which has a constant size for the same reason.</p>
<p>Let’s once again turn to OkHttp for a Kotlin example of this pattern.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">private</span> <span class="k">fun</span> <span class="nf">decodeIpv6</span><span class="p">(</span><span class="n">input</span><span class="p">:</span> <span class="nc">String</span><span class="p">,</span> <span class="n">pos</span><span class="p">:</span> <span class="nc">Int</span><span class="p">,</span> <span class="n">limit</span><span class="p">:</span> <span class="nc">Int</span><span class="p">):</span> <span class="nc">InetAddress</span><span class="p">?</span> <span class="p">{</span>
<span class="kd">val</span> <span class="py">address</span> <span class="p">=</span> <span class="nc">ByteArray</span><span class="p">(</span><span class="mi">16</span><span class="p">)</span>
<span class="kd">var</span> <span class="py">b</span> <span class="p">=</span> <span class="mi">0</span>
<span class="kd">var</span> <span class="py">i</span> <span class="p">=</span> <span class="n">pos</span>
<span class="k">while</span> <span class="p">(</span><span class="n">i</span> <span class="p"><</span> <span class="n">limit</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">b</span> <span class="p">==</span> <span class="n">address</span><span class="p">.</span><span class="n">size</span><span class="p">)</span> <span class="k">return</span> <span class="k">null</span> <span class="c1">// Too many groups.</span>
</code></pre></div></div>
<p>The use of <code class="highlighter-rouge">address.size</code> (which becomes a call to <code class="highlighter-rouge">length</code> in bytecode) prevents having to duplicate the 16 constant or extract it to a shared constant value. The downside is that each iteration of this parsing loop has resolve the array length as seen in output of <code class="highlighter-rouge">dx</code>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[00020c] OkHttpKt.decodeIpv6:(Ljava/lang/String;II)Ljava/net/InetAddress;
0000: const/16 v5, #int 16
0002: new-array v0, v5, [B
0004: const/4 v1, #int 0
0005: const/4 v2, #int 0
0006: if-ge v2, v8, 0036
0008: array-length v6, v0
0009: if-ne v1, v6, 000b
⋮
</code></pre></div></div>
<p>The constant 16 is loaded into register v5 at bytecode index 0000 which is used as the array size at index 0002. The resulting array reference is stored in register v0. The loop then starts at index 0006 with the <code class="highlighter-rouge">i < limit</code> comparison. Inside the loop, v0’s array length is loaded into v6 at index 0008 to be tested in the <code class="highlighter-rouge">if</code> at index 0009.</p>
<p>D8 recognizes that the <code class="highlighter-rouge">length</code> lookup is being done on an array reference which does not change and whose size is known at compile-time.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> [00020c] OkHttpKt.decodeIpv6:(Ljava/lang/String;II)Ljava/net/InetAddress;
0000: const/16 v5, #int 16
0002: new-array v0, v5, [B
0004: const/4 v1, #int 0
0005: const/4 v2, #int 0
0006: if-ge v2, v8, 0036
<span class="gd">-0008: array-length v6, v0
-0009: if-ne v1, v6, 000b
</span><span class="gi">+0009: if-ne v1, v5, 000b
</span> ⋮
</code></pre></div></div>
<p>The call to <code class="highlighter-rouge">array-length</code> is removed and the <code class="highlighter-rouge">if</code> is rewritten to re-use register v5 which is the size that was used to create the array.</p>
<p>On its own this pattern is not overly common. Once again it plays well when R8 inlining comes into effect and a method checking <code class="highlighter-rouge">array.length</code> is inlined into a caller that declares a new array.</p>
<hr />
<p>Each of these optimizations are small. D8 can only perform an optimization when it has no externally-visible effect and does not change program behavior. That pretty much limits it to optimizations which occur inside of a single method body.</p>
<p>At runtime you cannot tell that a switch was rewitten to if/else conditionals. You cannot tell that a call to <code class="highlighter-rouge">length()</code> on a constant string was replaced with its equivalent constant value. You cannot tell that a call to <code class="highlighter-rouge">length</code> on an array initialized in the same method was replaced with the input size. Each of these optimizations (and the few others) that D8 is able to perform result in slightly smaller and more-efficient bytecode. And, of course, when you invoke the full power of R8, their impact is multiplied.</p>
<p>In the next post we’ll start to cover how D8 backports new APIs on existing types to work on older API levels.</p>
R8 Optimization: Enum Switch Maps2019-10-16T00:00:00+00:00https://jakewharton.com/r8-optimization-enum-switch-maps<blockquote>
<p>Note: This post is part of a series on D8 and R8, Android’s new dexer and optimizer, respectively. For an intro to D8 read <a href="/androids-java-8-support/">“Android’s Java 8 support”</a>. For an intro to R8 read <a href="/r8-optimization-staticization/">“R8 Optimization: Staticization”</a>.</p>
</blockquote>
<p>The previous post on R8 covered <a href="/r8-optimization-enum-ordinals-and-names/#ordinal">enum ordinals</a> which then allowed branch elimination to apply to a switch statement. In that post, the full bytecode for <code class="highlighter-rouge">switch</code> on an enum was omitted because there’s actually more to the optimization.</p>
<p>Let’s start with a simple enum and a switch over its contents in two separate source files (this will be important later).</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">enum</span> <span class="nc">Greeting</span> <span class="o">{</span>
<span class="no">FORMAL</span><span class="o">,</span> <span class="no">INFORMAL</span>
<span class="o">}</span>
</code></pre></div></div>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Main</span> <span class="o">{</span>
<span class="kd">static</span> <span class="nc">String</span> <span class="nf">greetingType</span><span class="o">(</span><span class="nc">Greeting</span> <span class="n">greeting</span><span class="o">)</span> <span class="o">{</span>
<span class="k">switch</span> <span class="o">(</span><span class="n">greeting</span><span class="o">)</span> <span class="o">{</span>
<span class="k">case</span> <span class="nl">FORMAL:</span> <span class="k">return</span> <span class="s">"formal"</span><span class="o">;</span>
<span class="k">case</span> <span class="nl">INFORMAL:</span> <span class="k">return</span> <span class="s">"informal"</span><span class="o">;</span>
<span class="k">default</span><span class="o">:</span> <span class="k">throw</span> <span class="k">new</span> <span class="nc">AssertionError</span><span class="o">();</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">...</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">greetingType</span><span class="o">(</span><span class="nc">Greeting</span><span class="o">.</span><span class="na">INFORMAL</span><span class="o">));</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>If we compile and run these files the output is as expected.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ javac Greeting.java Main.java
$ java -cp . Main
informal
</code></pre></div></div>
<p>The bytecode in <a href="/r8-optimization-enum-ordinals-and-names/#ordinal">the previous post</a> showed that the compiler produces a call to <code class="highlighter-rouge">ordinal()</code> which is then used in the switch. But if that was all that the compiler did, re-ordering the constants of <code class="highlighter-rouge">Greeting</code> would break the output of <code class="highlighter-rouge">Main</code>.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> enum Greeting {
<span class="gd">- FORMAL, INFORMAL
</span><span class="gi">+ INFORMAL, FORMAL
</span> }
</code></pre></div></div>
<p>After changing the constant order, we can recompile <em>only</em> <code class="highlighter-rouge">Greeting.java</code> and yet the application still produces the correct output.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ javac Greeting.java
$ java -cp . Main
informal
</code></pre></div></div>
<p>If the bytecode was only relying on the value of <code class="highlighter-rouge">ordinal()</code>, this code would have produced “formal”.</p>
<h3 id="into-the-bytecode">Into The Bytecode</h3>
<p>To understand how this works we can look at the Java bytecode of <code class="highlighter-rouge">greetingType</code>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ javap -c Main.class
class Main {
static java.lang.String greetingType(Greeting);
Code:
0: getstatic #2 // Field Main$1.$SwitchMap$Greeting:[I
3: aload_0
4: invokevirtual #3 // Method Greeting.ordinal:()I
7: iaload
8: lookupswitch {
1: 36
2: 39
default: 42
}
36: ldc #4 // String formal
38: areturn
39: ldc #5 // String informal
41: areturn
42: new #6 // class java/lang/AssertionError
45: dup
46: invokespecial #7 // Method java/lang/AssertionError."<init>":()V
49: athrow
}
</code></pre></div></div>
<p>Let’s break the contents down. The first bytecode of this method has a lot of information to unpack:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>0: getstatic #2 // Field Main$1.$SwitchMap$Greeting:[I
</code></pre></div></div>
<p>This looks up a static field on the class <code class="highlighter-rouge">Main$1</code> with the name <code class="highlighter-rouge">$SwitchMap$Greeting</code> and the type <code class="highlighter-rouge">int[]</code>. We obviously did not write this class or field, so it must have been generated automatically by <code class="highlighter-rouge">javac</code>.</p>
<p>The next two bytecodes perform the call to <code class="highlighter-rouge">ordinal()</code> on the method argument.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>3: aload_0
4: invokevirtual #3 // Method Greeting.ordinal:()I
</code></pre></div></div>
<p>Java bytecode is stack-based, so the <code class="highlighter-rouge">int[]</code> result of <code class="highlighter-rouge">getstatic</code> and the <code class="highlighter-rouge">int</code> value of <code class="highlighter-rouge">ordinal()</code> both remain on the stack. (If you don’t understand how a stack-based machine works, you can watch <a href="/sinking-your-teeth-into-bytecode/">this presentation</a> for an introduction.) The next instruction uses that <code class="highlighter-rouge">int[]</code> and <code class="highlighter-rouge">int</code> as its operands.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>7: iaload
</code></pre></div></div>
<p>This “integer array load” instruction looks up a value in the <code class="highlighter-rouge">int[]</code> at the index returned by <code class="highlighter-rouge">ordinal()</code>. The rest of the bytecodes of the method are a “normal” switch statement which uses the value from the array as its input.</p>
<h3 id="switch-maps">Switch Maps</h3>
<p>It’s pretty clear that this <code class="highlighter-rouge">$SwitchMap$Greeting</code> array is the mechanism which allows our code to continue to work despite the ordinals changing their value. So how does it work?</p>
<p>When compiled, each <code class="highlighter-rouge">case</code> of the <code class="highlighter-rouge">switch</code> is assigned one-based index. The <code class="highlighter-rouge">default</code> branch is assigned zero.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">switch</span> <span class="o">(</span><span class="n">greeting</span><span class="o">)</span> <span class="o">{</span>
<span class="k">case</span> <span class="nl">FORMAL:</span> <span class="o">...</span> <span class="c1">// <-- index 1</span>
<span class="k">case</span> <span class="nl">INFORMAL:</span> <span class="o">...</span> <span class="c1">// <-- index 2</span>
<span class="k">default</span><span class="o">:</span> <span class="o">...</span> <span class="c1">// <-- index 0</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The <code class="highlighter-rouge">$SwitchMap$Greeting</code> array is populated at runtime in the static initializer of <code class="highlighter-rouge">Main$1</code>. The empty <code class="highlighter-rouge">int[]</code> is created first and assigned to the <code class="highlighter-rouge">$SwitchMap$Greeting</code> field.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>0: invokestatic #1 // Method Greeting.values:()[LGreeting;
3: arraylength
4: newarray int
6: putstatic #2 // Field $SwitchMap$Greeting:[I
</code></pre></div></div>
<p>The length of this array is the same as the number of constants (which might not match the number of <code class="highlighter-rouge">case</code> blocks). This is important since ordinals are used as an index into this array.</p>
<p>The next bytecodes are repeated for each constant used in the switch statement.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> 9: getstatic #2 // Field $SwitchMap$Greeting:[I
12: getstatic #3 // Field Greeting.FORMAL:LGreeting;
15: invokevirtual #4 // Method Greeting.ordinal:()I
18: iconst_1
19: iastore
</code></pre></div></div>
<p>The ordinal of <code class="highlighter-rouge">FORMAL</code>, the first <code class="highlighter-rouge">case</code> subject, is used as the offset in the array where its corresponding switch index value of 1 is stored. The same is done for the ordinal of <code class="highlighter-rouge">INFORMAL</code> and the value 2. This <code class="highlighter-rouge">int[]</code> effectively creates a map from the ordinals which may change to a fixed set of integer values which will not.</p>
<p><a href="/static/post-image/switch-map@2x.png">
<img class="center-block" src="/static/post-image/switch-map.png" srcset="/static/post-image/switch-map.png 1x, /static/post-image/switch-map@2x.png 2x" alt="Diagram showing the switch map working when the ordinals are changed." />
</a></p>
<p>By using this map, the switch statement can remain stable even when we re-arrange the constants of <code class="highlighter-rouge">Greeting</code>.</p>
<h3 id="the-optimization">The Optimization</h3>
<p>The switch map indirection created by <code class="highlighter-rouge">javac</code> is useful when the enum may be recompiled separately from the callers. Android applications are packaged as a single unit, so the indirection is nothing but wasted binary size and runtime overhead.</p>
<p>Running D8 on the class files from above shows that the indirection is maintained.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ java -jar $R8_HOME/build/libs/d8.jar \
--lib $ANDROID_HOME/platforms/android-29/android.jar \
--release \
--output . \
*.class
$ $ANDROID_HOME/build-tools/29.0.2/dexdump -d classes.dex
⋮
[00040c] Main.greetingType:(LGreeting;)Ljava/lang/String;
0000: sget-object v0, LMain$1;.$SwitchMap$Greeting:[I
0002: invoke-virtual {v1}, LGreeting;.ordinal:()I
0005: move-result v1
0006: aget v1, v0, v1
0008: packed-switch v1, 00000024
⋮
</code></pre></div></div>
<p>R8, however, performs whole-program analysis and optimization. There’s no point for it to retain this indirection since the enum cannot change independently of the switch.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> [00040c] Main.greetingType:(LGreeting;)Ljava/lang/String;
<span class="gd">-0000: sget-object v0, LMain$1;.$SwitchMap$Greeting:[I
</span> 0000: invoke-virtual {v1}, LGreeting;.ordinal:()I
0003: move-result v1
<span class="gd">-0006: aget v1, v0, v1
</span> 0004: packed-switch v1, 00000024
</code></pre></div></div>
<p>The branches of the switch are rewritten to account for the fact that the input now uses the zero-based ordinal directly instead of the one-based values from the switch map. With the <code class="highlighter-rouge">Main$1</code> class and its array being no longer referenced, it is eliminated like normal dead code.</p>
<p>Only with this indirection removed can the <a href="/r8-optimization-enum-ordinals-and-names/">enum ordinal optimization</a> from the previous post result in eliminating the switch. Otherwise, the ordinal value would flow into the <code class="highlighter-rouge">int[]</code> as an index which is not safe to eliminate in the general case.</p>
<h4 id="kotlin">Kotlin</h4>
<p>An enum used in a Kotlin <code class="highlighter-rouge">when</code> will also produce a similar indirection for the same reasons.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">val</span> <span class="py">Greeting</span><span class="p">.</span><span class="n">type</span> <span class="k">get</span><span class="p">()</span> <span class="p">=</span> <span class="k">when</span> <span class="p">(</span><span class="k">this</span><span class="p">)</span> <span class="p">{</span>
<span class="nc">Greeting</span><span class="p">.</span><span class="nc">FORMAL</span> <span class="p">-></span> <span class="s">"formal"</span>
<span class="nc">Greeting</span><span class="p">.</span><span class="nc">INFORMAL</span> <span class="p">-></span> <span class="s">"informal"</span>
<span class="p">}</span>
</code></pre></div></div>
<p>When compiled, the Java bytecode shows a similar mechanism but with different names.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ javap -c MainKt
public final class MainKt {
public static final java.lang.String getType(Greeting);
Code:
0: aload_0
1: getstatic #21 // Field MainKt$WhenMappings.$EnumSwitchMapping$0:[I
4: swap
5: invokevirtual #27 // Method Greeting.ordinal:()I
8: iaload
9: tableswitch {
1: 36
2: 41
default: 46
}
⋮
</code></pre></div></div>
<p>The generated class is suffixed with <code class="highlighter-rouge">$WhenMappings</code> instead of an arbitrary integer and the array is named <code class="highlighter-rouge">$EnumSwitchMapping$0</code>.</p>
<p>R8 initially did not detect Kotlin mappings because of these slightly different names. Version 1.6 of R8 (included in AGP 3.6) will correctly detect and eliminate them.</p>
<hr />
<p>Switch map elimination is a nice win for binary size and runtime performance. More importantly, by removing an indirection between the input to a switch and its branching logic, other optimizations like turning calls to <code class="highlighter-rouge">ordinal()</code> into a constant can result in branch elimination.</p>
<p>More R8 optimization posts coming soon. Stay tuned!</p>
R8 Optimization: Enum Ordinals and Names2019-10-09T00:00:00+00:00https://jakewharton.com/r8-optimization-enum-ordinals-and-names<blockquote>
<p>Note: This post is part of a series on D8 and R8, Android’s new dexer and optimizer, respectively. For an intro to D8 read <a href="/androids-java-8-support/">“Android’s Java 8 support”</a>. For an intro to R8 read <a href="/r8-optimization-staticization/">“R8 Optimization: Staticization”</a>.</p>
</blockquote>
<p>Enums are (and have always been!) a recommended way to model a fixed set of constants. Most commonly an enum only provides a set of possible constants and nothing more. But being full classes, enums can also carry helper methods and fields (both instance and static) or even implement interfaces.</p>
<p>A common optimization for enums in tools that perform whole-program optimization is to replace simple occurrences (i.e., those which don’t have fields, methods, or interfaces) with integer values. However, there are other optimizations which are applicable to <em>all</em> enums that are still available.</p>
<h3 id="ordinal">Ordinal</h3>
<p>Each enum constant has an <code class="highlighter-rouge">ordinal()</code> which returns its position in the list of all constants. Since the ordinal range is always [0, N), it can be used for indexing into other zero-based data structures such as arrays or even bits. The most common usage is actually by the Java compiler itself for <code class="highlighter-rouge">switch</code> statements over enums.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">enum</span> <span class="nc">Greeting</span> <span class="o">{</span>
<span class="no">FORMAL</span> <span class="o">{</span>
<span class="nd">@Override</span> <span class="nc">String</span> <span class="nf">greet</span><span class="o">(</span><span class="nc">String</span> <span class="n">name</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="s">"Hello, "</span> <span class="o">+</span> <span class="n">name</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">},</span>
<span class="no">INFORMAL</span> <span class="o">{</span>
<span class="nd">@Override</span> <span class="nc">String</span> <span class="nf">greet</span><span class="o">(</span><span class="nc">String</span> <span class="n">name</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="s">"Hey "</span> <span class="o">+</span> <span class="n">name</span> <span class="o">+</span> <span class="sc">'!'</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">};</span>
<span class="kd">abstract</span> <span class="nc">String</span> <span class="nf">greet</span><span class="o">(</span><span class="nc">String</span> <span class="n">name</span><span class="o">);</span>
<span class="kd">static</span> <span class="nc">String</span> <span class="nf">type</span><span class="o">(</span><span class="nc">Greeting</span> <span class="n">greeting</span><span class="o">)</span> <span class="o">{</span>
<span class="k">switch</span> <span class="o">(</span><span class="n">greeting</span><span class="o">)</span> <span class="o">{</span>
<span class="k">case</span> <span class="nl">FORMAL:</span> <span class="k">return</span> <span class="s">"formal"</span><span class="o">;</span>
<span class="k">case</span> <span class="nl">INFORMAL:</span> <span class="k">return</span> <span class="s">"informal"</span><span class="o">;</span>
<span class="k">default</span><span class="o">:</span> <span class="k">throw</span> <span class="k">new</span> <span class="nc">AssertionError</span><span class="o">();</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The compiled bytecode reveals the hidden call to <code class="highlighter-rouge">ordinal()</code>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[000a34] Greeting.type:(LGreeting;)Ljava/lang/String;
0000: invoke-virtual {v1}, LGreeting;.ordinal:()I
0003: move-result v1
⋮
</code></pre></div></div>
<p>If we call this method with one of the constants, an opportunity for optimization presents itself.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">...</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="nc">Greeting</span><span class="o">.</span><span class="na">type</span><span class="o">(</span><span class="nc">Greeting</span><span class="o">.</span><span class="na">INFORMAL</span><span class="o">));</span>
<span class="o">}</span>
</code></pre></div></div>
<p>As this is the only usage of <code class="highlighter-rouge">type</code> in our whole application, R8 inlines the method.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[000b60] Greeter.main:([Ljava/lang/String;)V
0000: sget-object v1, Ljava/lang/System;.out:Ljava/io/PrintStream;
0002: sget-object v0, LGreeting;.INFORMAL:LGreeting;
0004: invoke-virtual {v0}, LGreeting;.ordinal:()I
0007: move-result v0
⋮
0047: invoke-virtual {v1, v2}, Ljava/io/PrintStream;.println:(Ljava/lang/String;)V
0050: return-void
</code></pre></div></div>
<p>Bytecode index 0002 looks up the <code class="highlighter-rouge">INFORMAL</code> enum constant and then 0004 - 0007 invokes its <code class="highlighter-rouge">oridinal()</code> method. This is now a wasteful operation since the ordinal of the constant is known at compile-time.</p>
<p>R8 detects when a constant lookup flows into a call to <code class="highlighter-rouge">ordinal()</code> and replaces the call and lookup with the correct integer value that the call would produce.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> [000b60] Greeter.main:([Ljava/lang/String;)V
0000: sget-object v1, Ljava/lang/System;.out:Ljava/io/PrintStream;
<span class="gd">-0002: sget-object v0, LGreeting;.INFORMAL:LGreeting;
-0004: invoke-virtual {v0}, LGreeting;.ordinal:()I
-0007: move-result v0
</span><span class="gi">+0002: const/4 v0, #int 1
</span> ⋮
0042: invoke-virtual {v1, v2}, Ljava/io/PrintStream;.println:(Ljava/lang/String;)V
0045: return-void
</code></pre></div></div>
<p>This constant value now flows into the <code class="highlighter-rouge">switch</code> statement which can be eliminated leaving only the desired branch.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> [000b60] Greeter.main:([Ljava/lang/String;)V
0000: sget-object v1, Ljava/lang/System;.out:Ljava/io/PrintStream;
<span class="gd">-0002: const/4 v0, #int 1
- ⋮
</span><span class="gi">+0002: const-string v0, "informal"
</span> 0004: invoke-virtual {v1, v0}, Ljava/io/PrintStream;.println:(Ljava/lang/String;)V
0007: return-void
</code></pre></div></div>
<p>Even though the language provides switches over an enum, it’s implementation is all based on integers from the ordinal values. It’s a simple optimization to replace calls to <code class="highlighter-rouge">ordinal()</code> on fixed constants, but it enables more advanced optimizations like branch elimination to apply where they otherwise could not.</p>
<h3 id="name">Name</h3>
<p>In addition to <code class="highlighter-rouge">ordinal()</code>, each enum constant exposes its declared name through the <code class="highlighter-rouge">name()</code> method. The <code class="highlighter-rouge">toString()</code> will also return the declared name by default, but since that method can be overridden it’s important to have a distinct <code class="highlighter-rouge">name()</code>.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">enum</span> <span class="nc">Greeting</span> <span class="o">{</span>
<span class="no">FORMAL</span> <span class="o">{</span> <span class="cm">/* … */</span> <span class="o">},</span>
<span class="no">INFORMAL</span> <span class="o">{</span> <span class="cm">/* … */</span> <span class="o">};</span>
<span class="kd">abstract</span> <span class="nc">String</span> <span class="nf">greet</span><span class="o">(</span><span class="nc">String</span> <span class="n">name</span><span class="o">);</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="nc">String</span> <span class="nf">toString</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="s">"Greeting("</span> <span class="o">+</span> <span class="n">name</span><span class="o">().</span><span class="na">toLowercase</span><span class="o">(</span><span class="no">US</span><span class="o">)</span> <span class="o">+</span> <span class="sc">')'</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The value of <code class="highlighter-rouge">name()</code> is sometimes used for display, logging, or serialization.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">static</span> <span class="kt">void</span> <span class="nf">printGreeting</span><span class="o">(</span><span class="nc">Greeting</span> <span class="n">greeting</span><span class="o">,</span> <span class="nc">String</span> <span class="n">name</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">greeting</span><span class="o">.</span><span class="na">name</span><span class="o">()</span> <span class="o">+</span> <span class="s">": "</span> <span class="o">+</span> <span class="n">greeting</span><span class="o">.</span><span class="na">greet</span><span class="o">(</span><span class="n">name</span><span class="o">));</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">...</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
<span class="n">printGreeting</span><span class="o">(</span><span class="nc">Greeting</span><span class="o">.</span><span class="na">FORMAL</span><span class="o">,</span> <span class="s">"Jake"</span><span class="o">);</span>
<span class="o">}</span>
</code></pre></div></div>
<p>This program prints “FORMAL: Hello, Jake” when run. Once again, by virtue of only being called from one place, R8 inlines <code class="highlighter-rouge">printGreeting</code> into <code class="highlighter-rouge">main</code>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[000474] Greeting.main:([Ljava/lang/String;)V
0000: sget-object v3, LGreeting;.FORMAL:LGreeting;
0002: sget-object v0, Ljava/lang/System;.out:Ljava/io/PrintStream;
0004: new-instance v1, Ljava/lang/StringBuilder;
0006: invoke-direct {v1}, Ljava/lang/StringBuilder;.<init>:()V
0009: invoke-virtual {v3}, LGreeting;.name:()Ljava/lang/String;
000c: move-result-object v2
⋮
0022: invoke-virtual {v1, v2}, Ljava/io/PrintStream;.println:(Ljava/lang/String;)V
0025: return-void
</code></pre></div></div>
<p>Bytecode index 0000 looks up the <code class="highlighter-rouge">FORMAL</code> enum constant and then 0009 - 000c invokes its <code class="highlighter-rouge">name()</code> method. Just like <code class="highlighter-rouge">ordinal()</code>, this is a wasteful operation as the name of the constant is known at compile-time.</p>
<p>R8 again detects when a constant enum lookup flows into a call to <code class="highlighter-rouge">name()</code> and replaces the call and lookup with a string constant. If you read the <a href="/the-economics-of-generated-code/#string-duplication">economics of generated code</a> post, it talked about the cost of generating new string constants. Thankfully, because these strings share their name with the name of the enum constant, we do not pay for a new string.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> [000474] Greeting.main:([Ljava/lang/String;)V
0000: sget-object v3, LGreeting;.FORMAL:LGreeting;
0002: sget-object v0, Ljava/lang/System;.out:Ljava/io/PrintStream;
0004: new-instance v1, Ljava/lang/StringBuilder;
0006: invoke-direct {v1}, Ljava/lang/StringBuilder;.<init>:()V
<span class="gd">-0009: invoke-virtual {v3}, LGreeting;.name:()Ljava/lang/String;
-000c: move-result-object v2
</span><span class="gi">+0009: const-string v2, "FORMAL"
</span> ⋮
0020: invoke-virtual {v1, v2}, Ljava/io/PrintStream;.println:(Ljava/lang/String;)V
0023: return-void
</code></pre></div></div>
<p>The lookup at bytecode index 0000 still occurs because the code needs to invoke the <code class="highlighter-rouge">greet</code> method, but the call to <code class="highlighter-rouge">name()</code> was eliminated.</p>
<p>This optimization won’t enable other large optimizations like branch elimination to apply. But, since it produces a string, any <a href="/r8-optimization-string-constant-operations/">string operations</a> that are done on the result of the <code class="highlighter-rouge">name()</code> call may also be performed at compile-time.</p>
<p>For enums without a <code class="highlighter-rouge">toString()</code> override, this optimization will also apply to calls to <code class="highlighter-rouge">toString()</code> which defaults to being the same as <code class="highlighter-rouge">name()</code>.</p>
<hr />
<p>Both of these enum optimizations are small and really only work in the context of other R8 optimizations. Although, if it wasn’t clear in this series by now, that’s how most of these optimizations achieve their true power.</p>
<p>So far in this series I chose to highlight optimizations based on having found bugs in them or sometimes even suggesting them myself through the R8 issue tracker. But the two optimizations in this post are somewhat special because I actually managed to contribute these myself! I suspect we won’t see much else of my contribution in the series, but it feels good to have at least played a small part.</p>
<p>In the next post we’ll come back to the enum ordinal optimization because switch statements on enums are far more complicated than they seem. Stay tuned!</p>
R8 Optimization: Class Reflection and Forced Inlining2019-09-25T00:00:00+00:00https://jakewharton.com/r8-optimization-class-reflection-and-forced-inlining<blockquote>
<p>Note: This post is part of a series on D8 and R8, Android’s new dexer and optimizer, respectively. For an intro to D8 read <a href="/androids-java-8-support/">“Android’s Java 8 support”</a>. For an intro to R8 read <a href="/r8-optimization-staticization/">“R8 Optimization: Staticization”</a>.</p>
</blockquote>
<p>The previous post on R8 covered <a href="/r8-optimization-method-outlining/">method outlining</a> which automatically de-duplicated code. This was actually a detour from what I had promised was next at the end of the <a href="/r8-optimization-class-constant-operations/">class constant operations</a> post which preceded it. So let’s get back on track.</p>
<p>Class constant operations allow R8 to take calls such as <code class="highlighter-rouge">MyActivity.class.getSimpleName()</code> and replace it with the string literal <code class="highlighter-rouge">"MyActivity"</code>. This was presented in the context of log tags, where you might write that expression instead of the string literal so that the tag always reflects the actual class name, even after obfuscation. This works great in a static context where the <code class="highlighter-rouge">MyActivity.class</code> literal is fixed, but it does not work when used on an instance.</p>
<h3 id="instance-reflection">Instance reflection</h3>
<p>When dealing with an instance, the <code class="highlighter-rouge">Class</code> reference is obtained by calling <code class="highlighter-rouge">getClass()</code> instead of a <code class="highlighter-rouge">MyActivity.class</code> literal. This operation is not terribly expensive, but it is still a form of reflection.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">MyActivity</span> <span class="kd">extends</span> <span class="nc">Activity</span> <span class="o">{</span>
<span class="nd">@Override</span> <span class="kt">void</span> <span class="nf">onCreate</span><span class="o">(</span><span class="nc">Bundle</span> <span class="n">savedInstanceState</span><span class="o">)</span> <span class="o">{</span>
<span class="kd">super</span><span class="o">.</span><span class="na">onCreate</span><span class="o">(</span><span class="n">savedInstanceState</span><span class="o">);</span>
<span class="nc">String</span> <span class="n">name</span> <span class="o">=</span> <span class="k">this</span><span class="o">.</span><span class="na">getClass</span><span class="o">().</span><span class="na">getSimpleName</span><span class="o">();</span>
<span class="nc">Log</span><span class="o">.</span><span class="na">e</span><span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="s">"Hello!"</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The <code class="highlighter-rouge">getClass()</code> API is just a normal method on every <code class="highlighter-rouge">Object</code> and appears as a normal <code class="highlighter-rouge">invoke-virtual</code> in bytecode.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[0003d0] MyActivity.onCreate:(Landroid/os/Bundle;)V
0000: invoke-super {v1, v2}, Landroid/app/Activity;.onCreate:(Landroid/os/Bundle;)V
0003: invoke-virtual {v1}, Ljava/lang/Object;.getClass:()Ljava/lang/Class;
0006: move-result-object v2
0007: invoke-virtual {v2}, Ljava/lang/Class;.getSimpleName:()Ljava/lang/String;
000a: move-result-object v2
</code></pre></div></div>
<p>Since R8 is performing whole-program analysis, it knows that there are no subtypes of <code class="highlighter-rouge">MyActivity</code> even though it’s not marked as <code class="highlighter-rouge">final</code>. As a result, it can replace calls to <code class="highlighter-rouge">this.getClass()</code> with <code class="highlighter-rouge">MyActivity.class</code>.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> [000170] MyActivity.onCreate:(Landroid/os/Bundle;)V
0000: invoke-super {v1, v2}, Landroid/app/Activity;.onCreate:(Landroid/os/Bundle;)V
<span class="gd">-0003: invoke-virtual {v1}, Ljava/lang/Object;.getClass:()Ljava/lang/Class;
-0006: move-result-object v2
</span><span class="gi">+0003: const-class v2, Lcom/example/MyActivity;
</span> 0005: invoke-virtual {v2}, Ljava/lang/Class;.getSimpleName:()Ljava/lang/String;
0008: move-result-object v2
</code></pre></div></div>
<p>Beyond that, the <code class="highlighter-rouge">Class<?></code> reference immediately flows into a call to <code class="highlighter-rouge">getSimpleName()</code>. Thus, the optimization covered in <a href="/r8-optimization-class-constant-operations/">the previous post</a> can now apply producing only the simple constant string.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> 0000: invoke-super {v1, v2}, Landroid/app/Activity;.onCreate:(Landroid/os/Bundle;)V
<span class="gd">-0003: const-class v2, Lcom/example/MyActivity;
-0005: invoke-virtual {v2}, Ljava/lang/Class;.getSimpleName:()Ljava/lang/String;
-0008: move-result-object v2
</span><span class="gi">+0003: const-string v2, "MyActivity"
</span></code></pre></div></div>
<p>But how often do you write <code class="highlighter-rouge">this.getClass()</code> where the class is known unequivocally?</p>
<p>In keeping with the example of logging, let’s look at a hypothetical library which accepts an <code class="highlighter-rouge">Activity</code> and an optional name for use with logging.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">SomeLibrary</span> <span class="o">{</span>
<span class="kd">static</span> <span class="nc">SomeLibrary</span> <span class="nf">create</span><span class="o">(</span><span class="nc">Activity</span> <span class="n">activity</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="nf">create</span><span class="o">(</span><span class="n">activity</span><span class="o">,</span> <span class="n">activity</span><span class="o">.</span><span class="na">getClass</span><span class="o">().</span><span class="na">getSimpleName</span><span class="o">());</span>
<span class="o">}</span>
<span class="kd">static</span> <span class="nc">SomeLibrary</span> <span class="nf">create</span><span class="o">(</span><span class="nc">Activity</span> <span class="n">activity</span><span class="o">,</span> <span class="nc">String</span> <span class="n">name</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="nf">SomeLibrary</span><span class="o">(</span><span class="n">activity</span><span class="o">,</span> <span class="n">name</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">private</span> <span class="nf">SomeLibrary</span><span class="o">(</span><span class="nc">Activity</span> <span class="n">activity</span><span class="o">,</span> <span class="nc">String</span> <span class="n">name</span><span class="o">)</span> <span class="o">{</span>
<span class="c1">// ...</span>
<span class="o">}</span>
<span class="kt">void</span> <span class="nf">doSomething</span><span class="o">()</span> <span class="o">{</span>
<span class="nc">Log</span><span class="o">.</span><span class="na">d</span><span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="s">"Starting work!"</span><span class="o">);</span>
<span class="c1">// ...</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>When a name is not supplied, it is inferred from the activity class name using <code class="highlighter-rouge">getClass().getSimpleName()</code>. Since the input is not a fixed class literal, this cannot be replaced with a string at compile-time.</p>
<p>Calling this from an activity is straightforward and reminiscent of a few popular libraries.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">MyActivity</span> <span class="kd">extends</span> <span class="nc">Activity</span> <span class="o">{</span>
<span class="kd">private</span> <span class="nc">SomeLibrary</span> <span class="n">library</span><span class="o">;</span>
<span class="nd">@Override</span> <span class="kt">void</span> <span class="nf">onCreate</span><span class="o">(</span><span class="nc">Bundle</span> <span class="n">savedInstanceState</span><span class="o">)</span> <span class="o">{</span>
<span class="kd">super</span><span class="o">.</span><span class="na">onCreate</span><span class="o">(</span><span class="n">savedInstanceState</span><span class="o">);</span>
<span class="n">library</span> <span class="o">=</span> <span class="nc">SomeLibrary</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="k">this</span><span class="o">);</span>
<span class="o">}</span>
<span class="nd">@Override</span> <span class="kt">void</span> <span class="nf">onResume</span><span class="o">()</span> <span class="o">{</span>
<span class="n">library</span><span class="o">.</span><span class="na">doSomething</span><span class="o">();</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The inlining of method bodies has been a staple in previous R8 posts as it often unlocks optimizations that otherwise would not apply. This example is no different in that regard, but it <em>is</em> different because the <code class="highlighter-rouge">create(Activity)</code> method is too large to be inlined normally. The three method calls to <code class="highlighter-rouge">getClass()</code>, <code class="highlighter-rouge">getSimpleName()</code>, and the <code class="highlighter-rouge">create()</code> overload, along with specifying the arguments to those methods, exceeds the maximum allowed method body size for inline candidates.</p>
<h3 id="inlining-by-force">Inlining by force</h3>
<p>R8 advertises its configuration rules as being compatible with those documented for ProGuard, the tool it’s meant to replace. But aside from honoring what ProGuard supports, it does have a undocumented rules of its own. An example of this was shown in the <a href="/r8-optimization-value-assumption/">value assumption</a> post (and ProGuard has since come to add support for that rule!). While undocumented, this rule is supported by R8.</p>
<p>Another undocumented, R8-specific rule can help guide inlinining is <code class="highlighter-rouge">-alwaysinline</code>. This directive overrides the limitations of normal inlining to inline method bodies which might not otherwise be considered. Unfortunately, this rule is undocumented for a very good reason: it is completely unsupported and supposed to be for testing-purposes only.</p>
<p>By using <code class="highlighter-rouge">-alwaysinline</code>, the <code class="highlighter-rouge">create(Activity)</code> method can be forced to be inlined.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-alwaysinline class com.example.SomeLibrary {
static void create(android.app.Activity);
}
</code></pre></div></div>
<p>This causes the <code class="highlighter-rouge">getClass().getSimpleName()</code> calls to be moved from the library code to each call site.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> @Override void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
<span class="gd">- library = SomeLibrary.create(this);
</span><span class="gi">+ library = SomeLibrary.create(this, this.getClass().getSimpleName());
</span> }
</code></pre></div></div>
<p>As a result, we’ve created the above scenario where the enclosing class is known at compile time. It will be replaced with the <code class="highlighter-rouge">MyActivity.class</code> class literal which is then quickly replaced with the <code class="highlighter-rouge">"MyActivity"</code> string literal.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> @Override void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
<span class="gd">- library = SomeLibrary.create(this, this.getClass().getSimpleName());
</span><span class="gi">+ library = SomeLibrary.create(this, "MyActivity");
</span> }
</code></pre></div></div>
<p>Once again we see the power of successive optimizations applying. No more reflection!</p>
<p>Unlike previous posts where inlining happened automatically, the unsupported <code class="highlighter-rouge">-alwaysinline</code> directive forced this behavior in R8. Inlining should only be forced like this when you know that a subsequent optimization will apply to offset the bytecode impact. In this example, there is a chance that the instance cannot be determined at compile-time and we end up slightly bloating the bytecode. And, of course, the unsupported nature of the rule means it may change or disappear at any time. For a stable solution, Kotlin’s <code class="highlighter-rouge">inline</code> function modifier has the same effect, but only for Kotlin callers.</p>
<hr />
<p>Replacing calls to <code class="highlighter-rouge">getClass()</code> with a class literal is a very small optimization. It saves only four bytes when inlined, but its greatest contribution is enabling other optimizations to apply. Subsequent calls to methods like <code class="highlighter-rouge">getSimpleName()</code> can now be eliminated which then opens up <a href="/r8-optimization-string-constant-operations/">string optimizations</a> to potentially apply.</p>
<p>In future R8 posts we’ll come back to this <code class="highlighter-rouge">getClass()</code> optimization and others which it enables. But for now, there’s a lot of other R8 optimizations that I want to cover without promising a specific topic next, so stay tuned.</p>
Calculating the true impact of zip file entries2019-09-20T00:00:00+00:00https://jakewharton.com/calculating-zip-file-entry-true-impact<p>How can we determine the impact of each entry on a zip file’s size? It seems like a trivial problem, but things quickly don’t add up.</p>
<p>There’s three built-in ways to read information about the contents of zip file in Java:</p>
<ol>
<li>Mount the zip as a <code class="highlighter-rouge">FileSystem</code> using <code class="highlighter-rouge">FileSystems.newFileSystem</code> and then access its contents using <code class="highlighter-rouge">Path</code>s.</li>
<li>Open it with <code class="highlighter-rouge">ZipInputStream</code> for a one-shot iteration over the zip entries.</li>
<li>Open it with <code class="highlighter-rouge">ZipFile</code> for random access to the zip entries.</li>
</ol>
<p>The first mechanism is extremely convenient. It allows interacting with the contents of a zip file using the same APIs as normal files. Unfortunately, by virtue of being exposed like regular files, you only have one way to check their size: <code class="highlighter-rouge">Files.size(Path)</code>. This delegates to an API called <code class="highlighter-rouge">BasicFileAttributes.size()</code> which returns size of the file contents. While there is a <code class="highlighter-rouge">ZipFileAttributes.compressedSize()</code> for returning the size of the compressed contents, it’s internal to the JDK and not available for our use.</p>
<p>The other two mechanisms,<code class="highlighter-rouge">ZipInputStream</code> and <code class="highlighter-rouge">ZipFile</code>, both expose entries using the <code class="highlighter-rouge">ZipEntry</code> type. These being zip-centric APIs, many of the properties of the zip file format are directly available. Notably for our use case, there’s a <code class="highlighter-rouge">getCompressedSize()</code> method.</p>
<p>Problem solved? Not exactly…</p>
<p>If you sum the compressed size of all entries in a zip the result will not equal the size of the zip file. This isn’t entirely unexpected. After all, the zip file format surely requires additional metadata to track per-entry information like the relative path of each compressed file.</p>
<p>So if we’re looking to calculate the <em>actual</em> size impact of an entry on the final zip, can we do it?</p>
<h3 id="zip-file-format">Zip file format</h3>
<p>An overview of the zip file format specification can be <a href="https://en.wikipedia.org/wiki/Zip_(file_format)#Structure">found on Wikipedia</a>. It consists of a list of entries which are each defined as header followed by the compressed data (whose length is specified in the header). Finally, at the end, there is a central directory which lists all of the entries available in the file.</p>
<p><a href="/static/post-image/zip_layout@2x.png">
<img src="/static/post-image/zip_layout.png" srcset="/static/post-image/zip_layout.png 1x, /static/post-image/zip_layout@2x.png 2x" alt="Diagram showing the zip file format as previously described." />
</a></p>
<p>A slight tangent: Given this format, it’s pretty obvious how <code class="highlighter-rouge">ZipInputStream</code> and <code class="highlighter-rouge">ZipFile</code> work. The former simply iterates forward through the bytes reading each entry as it comes. The latter parses the central directory at the end and then jumps to the offset of whichever entry you request.</p>
<p>Back on our problem, <code class="highlighter-rouge">ZipEntry.getCompressedSize()</code> is only exposing the length of compressed data (pictured as the blue <code class="highlighter-rouge"><data></code> blocks). However, the header for each entry and the record in the central directory also contribute to the overall size impact. Thus, to get the real value, we need to be able to calculate the size of those two things.</p>
<h4 id="zip-entry-header">Zip entry header</h4>
<p>The header for each entry is defined as follows:</p>
<style type="text/css">
th,td { padding-right: 15px; padding-bottom: 5px; }
table { margin-bottom: 15px; }
</style>
<table>
<thead>
<tr>
<th>Offset</th>
<th>Size</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td>4</td>
<td>Local file header signature</td>
</tr>
<tr>
<td>…</td>
<td>…</td>
<td>…</td>
</tr>
<tr>
<td>26</td>
<td>2</td>
<td>File name length (n)</td>
</tr>
<tr>
<td>28</td>
<td>2</td>
<td>Extra field length (m)</td>
</tr>
<tr>
<td>30</td>
<td>n</td>
<td>File name</td>
</tr>
<tr>
<td>30+n</td>
<td>m</td>
<td>Extra field</td>
</tr>
</tbody>
</table>
<p>Here we can see that the size of the header will be a fixed 30 bytes plus the length of <code class="highlighter-rouge">ZipEntry.getName()</code> (as UTF-8 bytes) plus the length of <code class="highlighter-rouge">ZipEntry.getExtra()</code> (which returns opaque bytes).</p>
<p>There is also an optional trailer which can be either 12 or 16 bytes. This is only present when a specific bit in one of the fields of the header is set. Unfortunately, the field which contains the bit is not exposed in the API of <code class="highlighter-rouge">ZipEntry</code>, and so we cannot include it in the calculation. Thankfully, this seems infrequently used.</p>
<h4 id="central-directory-record">Central directory record</h4>
<p>The central directory is a list of records for each file followed by a single end-of-directory record.</p>
<p>The record for each entry is defined as follows:</p>
<table>
<thead>
<tr>
<th>Offset</th>
<th>Size</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td>4</td>
<td>Central directory file header signature</td>
</tr>
<tr>
<td>…</td>
<td>…</td>
<td>…</td>
</tr>
<tr>
<td>42</td>
<td>4</td>
<td>Relative offset of local file header.</td>
</tr>
<tr>
<td>46</td>
<td>n</td>
<td>File name</td>
</tr>
<tr>
<td>46+n</td>
<td>m</td>
<td>Extra field</td>
</tr>
<tr>
<td>46+n+m</td>
<td>k</td>
<td>File comment</td>
</tr>
</tbody>
</table>
<p>The size will be 46 bytes plus the length of <code class="highlighter-rouge">ZipEntry.getName()</code> plus the length of <code class="highlighter-rouge">ZipEntry.getExtra()</code> plus the length of <code class="highlighter-rouge">ZipEntry.getComment()</code> (as UTF-8 bytes).</p>
<p>The end-of-directory record is defined as follows:</p>
<table>
<thead>
<tr>
<th>Offset</th>
<th>Size</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td>4</td>
<td>End of central directory signature</td>
</tr>
<tr>
<td>…</td>
<td>…</td>
<td>…</td>
</tr>
<tr>
<td>20</td>
<td>2</td>
<td>Comment length (n)</td>
</tr>
<tr>
<td>22</td>
<td>n</td>
<td>Comment</td>
</tr>
</tbody>
</table>
<p>Its size is 22 bytes plus the length of <code class="highlighter-rouge">ZipFile.getComment()</code> (as UTF-8) bytes. <code class="highlighter-rouge">ZipInputStream</code>, since it only iterates forward over the entries, does not expose the zip comment.</p>
<h3 id="putting-it-all-together">Putting it all together</h3>
<p>With this knowledge of the zip file format we can now calculate a more accurate representation of the impact of each entry.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">static</span> <span class="kt">long</span> <span class="nf">entryImpactBytes</span><span class="o">(</span><span class="nc">ZipEntry</span> <span class="n">entry</span><span class="o">)</span> <span class="o">{</span>
<span class="kt">int</span> <span class="n">nameSize</span> <span class="o">=</span> <span class="n">entry</span><span class="o">.</span><span class="na">getName</span><span class="o">().</span><span class="na">getBytes</span><span class="o">(</span><span class="no">UTF_8</span><span class="o">).</span><span class="na">length</span><span class="o">;</span>
<span class="kt">int</span> <span class="n">extraSize</span> <span class="o">=</span> <span class="n">entry</span><span class="o">.</span><span class="na">getExtra</span><span class="o">()</span> <span class="o">!=</span> <span class="kc">null</span>
<span class="o">?</span> <span class="n">entry</span><span class="o">.</span><span class="na">getExtra</span><span class="o">().</span><span class="na">length</span>
<span class="o">:</span> <span class="mi">0</span><span class="o">;</span>
<span class="kt">int</span> <span class="n">commentSize</span> <span class="o">=</span> <span class="n">entry</span><span class="o">.</span><span class="na">getComment</span><span class="o">()</span> <span class="o">!=</span> <span class="kc">null</span>
<span class="o">?</span> <span class="n">entry</span><span class="o">.</span><span class="na">getComment</span><span class="o">().</span><span class="na">getBytes</span><span class="o">(</span><span class="no">UTF_8</span><span class="o">).</span><span class="na">length</span>
<span class="o">:</span> <span class="mi">0</span><span class="o">;</span>
<span class="c1">// Calculate the actual compressed size impact in the zip, not just compressed data size.</span>
<span class="c1">// See https://en.wikipedia.org/wiki/Zip_(file_format)#File_headers for details.</span>
<span class="k">return</span> <span class="n">entry</span><span class="o">.</span><span class="na">getCompressedSize</span><span class="o">()</span>
<span class="c1">// Local file header. There is no way of knowing whether a trailing data descriptor</span>
<span class="c1">// was present since the general flags field is not exposed, but it's unlikely.</span>
<span class="o">+</span> <span class="mi">30</span> <span class="o">+</span> <span class="n">nameSize</span> <span class="o">+</span> <span class="n">extraSize</span>
<span class="c1">// Central directory file header.</span>
<span class="o">+</span> <span class="mi">46</span> <span class="o">+</span> <span class="n">nameSize</span> <span class="o">+</span> <span class="n">extraSize</span> <span class="o">+</span> <span class="n">commentSize</span><span class="o">;</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Using this method, a sum of all entries will put you very close to the actual size of the zip file. All that’s left is to account for the end-of-directory record from the central directory.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">static</span> <span class="kt">int</span> <span class="nf">additionalBytes</span><span class="o">(</span><span class="nc">ZipFile</span> <span class="n">file</span><span class="o">)</span> <span class="o">{</span>
<span class="kt">int</span> <span class="n">commentSize</span> <span class="o">=</span> <span class="n">file</span><span class="o">.</span><span class="na">getComment</span><span class="o">()</span> <span class="o">!=</span> <span class="kc">null</span>
<span class="o">?</span> <span class="n">file</span><span class="o">.</span><span class="na">getComment</span><span class="o">().</span><span class="na">getBytes</span><span class="o">(</span><span class="no">UTF_8</span><span class="o">).</span><span class="na">length</span>
<span class="o">:</span> <span class="mi">0</span><span class="o">;</span>
<span class="k">return</span> <span class="mi">22</span> <span class="o">+</span> <span class="n">commentSize</span><span class="o">;</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Using these two functions, the sum total should now exactly match the size of the zip file.</p>
<p>There’s some small improvements to be had here if we want. For one, we don’t need to encode the name and comment as UTF-8 bytes only then to get its length. Libraries like <a href="https://guava.dev/releases/19.0/api/docs/com/google/common/base/Utf8.html#encodedLength(java.lang.CharSequence)">Guava</a> and <a href="https://square.github.io/okio/2.x/okio/okio/kotlin.-string/utf8-size/">Okio</a> provide methods for calculating the UTF-8 length directly on a <code class="highlighter-rouge">String</code>. Additionally, the zip format is so simple that you could write your own parser which included the file trailers in its calculation depending on how accurate you needed the numbers to be.</p>
<hr />
<p>This <code class="highlighter-rouge">entryImpactBytes</code> method can be useful for calculating how much a zip file size will change when an entry is added or removed. But it really shines when you have two versions of a zip file. For example, reducing the contents of one file by 100 bytes <em>and</em> removing 50 bytes from its name will result in a net change of -200 bytes (2 * name diff + content diff). If you were only using <code class="highlighter-rouge">ZipEntry.getCompressedSize()</code> to compute such a difference, the result would only show a change of -100 bytes.</p>
Exceptions and proxies and coroutines, oh my!2019-07-31T00:00:00+00:00https://jakewharton.com/exceptions-and-proxies-and-coroutines-oh-my<p>Checked exceptions are a concept that exist only in the Java compiler and are enforced only in source code. In Java bytecode and at runtime in the virtual machine you’re free to throw checked exceptions from anywhere regardless of whether they’re declared. At least, anywhere <em>except</em> from a instance created by a Java <code class="highlighter-rouge">Proxy</code>.</p>
<p>A <code class="highlighter-rouge">Proxy</code> creates instances of interfaces at runtime where a single callback intercepts every method call. Libraries like <a href="https://square.github.io/retrofit/">Retrofit</a> use proxies to create HTTP calls based on the annotations of interface methods. These methods tend to return promise-like objects such as RxJava’s <code class="highlighter-rouge">Single</code>, Guava’s <code class="highlighter-rouge">ListenableFuture</code>, or its own <code class="highlighter-rouge">Call</code> type.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// MyService.java</span>
<span class="kd">interface</span> <span class="nc">MyService</span> <span class="o">{</span>
<span class="nd">@GET</span><span class="o">(</span><span class="s">"/user/{id}"</span><span class="o">)</span>
<span class="nc">Call</span><span class="o"><</span><span class="nc">User</span><span class="o">></span> <span class="nf">user</span><span class="o">(</span><span class="nd">@Path</span><span class="o">(</span><span class="s">"id"</span><span class="o">)</span> <span class="kt">long</span> <span class="n">id</span><span class="o">);</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Retrofit recently added support for Kotlin coroutines’ <code class="highlighter-rouge">suspend</code> functions which behave a bit differently. Aside from the <code class="highlighter-rouge">suspend</code> modifier, the method signature otherwise <em>appears</em> synchronous.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// MyService.kt</span>
<span class="kd">interface</span> <span class="nc">MyService</span> <span class="p">{</span>
<span class="nd">@GET</span><span class="p">(</span><span class="s">"/user/{id}"</span><span class="p">)</span>
<span class="k">suspend</span> <span class="k">fun</span> <span class="nf">user</span><span class="p">(</span><span class="nd">@Path</span><span class="p">(</span><span class="s">"id"</span><span class="p">)</span> <span class="n">id</span><span class="p">:</span> <span class="nc">Long</span><span class="p">):</span> <span class="nc">User</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Kotlin does not require declaring checked exceptions. With Retrofit using a <code class="highlighter-rouge">Proxy</code> and performing a network call that may throw an <code class="highlighter-rouge">IOException</code>, you might expect to be required to declare <code class="highlighter-rouge">@Throws(IOException::class)</code> though. This isn’t actually required because the method signature gets rewritten by the Kotlin compiler to accept a <code class="highlighter-rouge">Continuation</code> parameter where both exceptions and results are forwarded.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Approximate Java for the compiled bytecode of MyService.kt:</span>
<span class="kd">interface</span> <span class="nc">MyService</span> <span class="o">{</span>
<span class="kt">void</span> <span class="nf">user</span><span class="o">(</span><span class="nd">@Path</span><span class="o">(</span><span class="s">"id"</span><span class="o">)</span> <span class="kt">long</span> <span class="n">id</span><span class="o">,</span> <span class="nc">Continuation</span><span class="o"><?</span> <span class="kd">super</span> <span class="nc">User</span><span class="o">></span> <span class="n">continuation</span><span class="o">);</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Despite rewriting the bytecode to be callback-based and Retrofit asynchronously invoking the <code class="highlighter-rouge">Continuation</code>, rare calls to this method were resulting in an <code class="highlighter-rouge">UndeclaredThrowableException</code>. This indicates a checked exception was somehow being <em>synchronously</em> thrown.</p>
<p>To understand why this was occurring and to craft a fix, we need to learn more about how coroutines work…</p>
<h3 id="coroutine-implementation-crash-course">Coroutine Implementation Crash Course</h3>
<p>The above approximation of the Kotlin <code class="highlighter-rouge">MyService</code> bytecode is inaccurate. While the <code class="highlighter-rouge">Continuation</code> parameter is the <em>primary</em> mechanism of delivering a success or error result, it’s not the only mechanism.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Exact Java equivalent of MyService.kt bytecode</span>
<span class="kd">interface</span> <span class="nc">MyService</span> <span class="o">{</span>
<span class="nc">Object</span> <span class="nf">user</span><span class="o">(</span><span class="nd">@Path</span><span class="o">(</span><span class="s">"id"</span><span class="o">)</span> <span class="kt">long</span> <span class="n">id</span><span class="o">,</span> <span class="nc">Continuation</span><span class="o"><?</span> <span class="kd">super</span> <span class="nc">User</span><span class="o">></span> <span class="n">continuation</span><span class="o">);</span>
<span class="o">}</span>
</code></pre></div></div>
<p><code class="highlighter-rouge">Object</code> is used as the return type because a <code class="highlighter-rouge">User</code> instance can be directly returned if available synchronously. Otherwise, the method returns the <a href="https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.coroutines.intrinsics/-c-o-r-o-u-t-i-n-e_-s-u-s-p-e-n-d-e-d.html">“coroutine suspended” marker object</a> to indicate suspension (where the result will be delivered to the <code class="highlighter-rouge">Continuation</code>).</p>
<p>This is one way that a checked exception could occur synchronously outside of Retrofit. When the method fails synchronously, the exception is allowed to propagate.</p>
<p>For asynchronous results, the Kotlin standard library provides the <code class="highlighter-rouge">suspendCoroutine</code> API.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">suspend</span> <span class="k">fun</span> <span class="nf">user</span><span class="p">(</span><span class="n">id</span><span class="p">:</span> <span class="nc">Long</span><span class="p">):</span> <span class="nc">User</span> <span class="p">{</span>
<span class="k">return</span> <span class="nf">suspendCoroutine</span> <span class="p">{</span> <span class="n">continuation</span> <span class="p">-></span>
<span class="n">executor</span><span class="p">.</span><span class="nf">execute</span> <span class="p">{</span>
<span class="n">continuation</span><span class="p">.</span><span class="nf">resume</span><span class="p">(</span><span class="nc">User</span><span class="p">(</span><span class="s">"jw"</span><span class="p">))</span>
<span class="c1">// or continuation.resumeWithException(IOException("broken"))</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This approximates to the following Java source:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="nc">Object</span> <span class="nf">user</span><span class="o">(</span><span class="kt">long</span> <span class="n">id</span><span class="o">,</span> <span class="nc">Continuation</span><span class="o"><?</span> <span class="kd">super</span> <span class="nc">User</span><span class="o">></span> <span class="n">continuation</span><span class="o">)</span> <span class="o">{</span>
<span class="c1">// code inside lambda that calls into 'continuation'</span>
<span class="k">return</span> <span class="no">COROUTINE_SUSPENDED</span><span class="o">;</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The marker object is returned up the stack which frees the thread to run other code. Once the continuation is invoked, our code will resume as soon as any thread is free again.</p>
<h3 id="retrofit-coroutine-implementation">Retrofit Coroutine Implementation</h3>
<p>Retrofit uses the <code class="highlighter-rouge">suspendCoroutine</code> API with its own <code class="highlighter-rouge">Callback</code> to suspend while the HTTP request is sent on a background thread.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">suspend</span> <span class="k">fun</span> <span class="p"><</span><span class="nc">T</span> <span class="p">:</span> <span class="nc">Any</span><span class="p">></span> <span class="nf">Call</span><span class="p"><</span><span class="nc">T</span><span class="p">>.</span><span class="nf">awaitResponse</span><span class="p">():</span> <span class="nc">Response</span><span class="p"><</span><span class="nc">T</span><span class="p">></span> <span class="p">{</span>
<span class="k">return</span> <span class="nf">suspendCoroutine</span> <span class="p">{</span> <span class="n">continuation</span> <span class="p">-></span>
<span class="nf">enqueue</span><span class="p">(</span><span class="kd">object</span> <span class="err">: </span><span class="nc">Callback</span><span class="p"><</span><span class="nc">T</span><span class="p">></span> <span class="p">{</span>
<span class="k">override</span> <span class="k">fun</span> <span class="nf">onResponse</span><span class="p">(</span><span class="n">call</span><span class="p">:</span> <span class="nc">Call</span><span class="p"><</span><span class="nc">T</span><span class="p">>,</span> <span class="n">response</span><span class="p">:</span> <span class="nc">Response</span><span class="p"><</span><span class="nc">T</span><span class="p">>)</span> <span class="p">{</span>
<span class="n">continuation</span><span class="p">.</span><span class="nf">resume</span><span class="p">(</span><span class="n">response</span><span class="p">)</span>
<span class="p">}</span>
<span class="k">override</span> <span class="k">fun</span> <span class="nf">onFailure</span><span class="p">(</span><span class="n">call</span><span class="p">:</span> <span class="nc">Call</span><span class="p"><</span><span class="nc">T</span><span class="p">>,</span> <span class="n">t</span><span class="p">:</span> <span class="nc">Throwable</span><span class="p">)</span> <span class="p">{</span>
<span class="n">continuation</span><span class="p">.</span><span class="nf">resumeWithException</span><span class="p">(</span><span class="n">t</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">})</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The implementation of <code class="highlighter-rouge">Call.enqueue</code> is very similar to the sample above which calls <code class="highlighter-rouge">executor.execute { .. }</code>. A thread pool picks up the <code class="highlighter-rouge">Call</code>, runs the request, and invokes the <code class="highlighter-rouge">Callback</code> when a reply is received.</p>
<p>It seems that Retrofit is not doing any work synchronously that would cause a checked exception. The stacktrace of the <code class="highlighter-rouge">UndeclaredThrowableException</code> even confirms that the work ran on the background <code class="highlighter-rouge">Executor</code>:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>java.lang.reflect.UndeclaredThrowableException
at ...
Caused by: java.net.UnknownHostException
at ...
at retrofit2.AsyncCall.execute(AsyncCall.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
</code></pre></div></div>
<p>Despite doing everything seemingly right, there’s still clearly a bug or we never would see the <code class="highlighter-rouge">UndeclaredThrowableException</code>.</p>
<h3 id="the-bug">The Bug</h3>
<p>There’s one behavior of <code class="highlighter-rouge">suspendCoroutine</code> that was not mentioned above which is designed to protect the execution stack. If the lambda passed to <code class="highlighter-rouge">suspendCoroutine</code> invokes the <code class="highlighter-rouge">Continuation</code> parameter <em>synchronously</em> then instead of calling the real <code class="highlighter-rouge">Continuation</code>, the value is intercepted and propagated synchronously.</p>
<p>Going back to the sample, removing the call to <code class="highlighter-rouge">executor.execute</code> would create this behavior.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">suspend</span> <span class="k">fun</span> <span class="nf">user</span><span class="p">(</span><span class="n">id</span><span class="p">:</span> <span class="nc">Long</span><span class="p">):</span> <span class="nc">User</span> <span class="p">{</span>
<span class="k">return</span> <span class="nf">suspendCoroutine</span> <span class="p">{</span> <span class="n">continuation</span> <span class="p">-></span>
<span class="n">continuation</span><span class="p">.</span><span class="nf">resume</span><span class="p">(</span><span class="nc">User</span><span class="p">(</span><span class="s">"jw"</span><span class="p">))</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Without stack protection, invoking the continuation like this could cause the caller’s code to resume <em>beneath</em> the current stack frame. This would lead to extremely deep call stacks which would eventually trigger a <code class="highlighter-rouge">StackOverflowError</code>.</p>
<p><code class="highlighter-rouge">suspendCoroutine</code> performs interception by wrapping the <code class="highlighter-rouge">Continuation</code>. Here is the approximated Java equivalent:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="nc">Object</span> <span class="nf">user</span><span class="o">(</span><span class="kt">long</span> <span class="n">id</span><span class="o">,</span> <span class="nc">Continuation</span><span class="o"><?</span> <span class="kd">super</span> <span class="nc">User</span><span class="o">></span> <span class="n">real</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">ContinuationImpl</span><span class="o"><?</span> <span class="kd">super</span> <span class="nc">User</span><span class="o">></span> <span class="n">continuation</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ContinuationImpl</span><span class="o">(</span><span class="n">real</span><span class="o">);</span>
<span class="c1">// code inside lambda that calls into 'continuation'</span>
<span class="k">return</span> <span class="n">continuation</span><span class="o">.</span><span class="na">getResult</span><span class="o">();</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The <code class="highlighter-rouge">getResult()</code> call will do one of three things:</p>
<ol>
<li>If <code class="highlighter-rouge">resume</code> was already called on <code class="highlighter-rouge">continuation</code>, return the value that was supplied.</li>
<li>If <code class="highlighter-rouge">resumeWithException</code>was already called on <code class="highlighter-rouge">continuation</code>, throw the exception that was supplied.</li>
<li>Otherwise, return <code class="highlighter-rouge">COROUTINE_SUSPENDED</code>. Future calls to <code class="highlighter-rouge">resume</code> and <code class="highlighter-rouge">resumeWithException</code> will forward to the <code class="highlighter-rouge">real</code> continuation.</li>
</ol>
<p>The behavior of case #2 provides a probable source of a checked exception being thrown synchronously which in turn would cause the <code class="highlighter-rouge">UndeclaredThrowableException</code>.</p>
<p>But this only explains the bug if the callback is invoked <em>before</em> the calling method is able to return. Since <code class="highlighter-rouge">enqueue</code> dispatches work to an <code class="highlighter-rouge">Executor</code> and immediately returns, the likelihood of this happening is zero. That is, at least, until you consider <a href="https://en.wikipedia.org/wiki/Preemption_(computing)">preemption</a>.</p>
<p>There are two threads here: the caller and the background worker. If we ignore the case where these execute on different CPU cores, a single core may preempt the caller thread to let the background worker make progress.</p>
<p><a href="/static/post-image/exception-proxy-coroutine-1@2x.png">
<img src="/static/post-image/exception-proxy-coroutine-1.png" srcset="/static/post-image/exception-proxy-coroutine-1.png 1x, /static/post-image/exception-proxy-coroutine-1@2x.png 2x" alt="Diagram showing the caller thread being preempted between the call to enqueue and returning and the worker thread invoking the continuation" />
</a></p>
<p>Occasionally that preemption will occur precisely between the <code class="highlighter-rouge">ContinuationImpl</code> creation (green) and the call to <code class="highlighter-rouge">getResult()</code> (red). If the background work is quick enough the continuation may be invoked (orange) before switching back. In this example, an exception is quickly thrown due to a failed DNS lookup that was cached.</p>
<h3 id="the-fix">The Fix</h3>
<p>Detecting this case in Retrofit is simple. When the Java-based implementation delegates to the <code class="highlighter-rouge">suspend fun</code> it captures checked exceptions with a <code class="highlighter-rouge">try</code>/<code class="highlighter-rouge">catch</code> block.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">try</span> <span class="o">{</span>
<span class="k">return</span> <span class="nc">KotlinExtensions</span><span class="o">.</span><span class="na">awaitResponse</span><span class="o">(</span><span class="n">call</span><span class="o">,</span> <span class="n">continuation</span><span class="o">);</span>
<span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">Exception</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
<span class="c1">// but now what?</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Invoking the <code class="highlighter-rouge">continuation</code> in the <code class="highlighter-rouge">catch</code> block is possible, but would defeat the stack protection of <code class="highlighter-rouge">suspendCoroutine</code> that caused this behavior in the first place. The current method call needs to be suspended before the exception is delivered. In Kotlin, this can be achieved with <code class="highlighter-rouge">yield()</code>.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">suspend</span> <span class="k">fun</span> <span class="nc">Exception</span><span class="p">.</span><span class="nf">yieldAndThrow</span><span class="p">():</span> <span class="nc">Nothing</span> <span class="p">{</span>
<span class="k">yield</span><span class="p">()</span>
<span class="k">throw</span> <span class="k">this</span>
<span class="p">}</span>
</code></pre></div></div>
<p>From Java this function which will always return <code class="highlighter-rouge">COROUTINE_SUSPENDED</code> because of <code class="highlighter-rouge">yield()</code>. The <code class="highlighter-rouge">continuation</code> will then receive the exception at the next available time on the current coroutine dispatcher.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">try</span> <span class="o">{</span>
<span class="k">return</span> <span class="nc">KotlinExtensions</span><span class="o">.</span><span class="na">awaitResponse</span><span class="o">(</span><span class="n">call</span><span class="o">,</span> <span class="n">continuation</span><span class="o">);</span>
<span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">Exception</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="nc">KotlinExtensions</span><span class="o">.</span><span class="na">yieldAndThrow</span><span class="o">(</span><span class="n">e</span><span class="o">,</span> <span class="n">continuation</span><span class="o">);</span>
<span class="o">}</span>
</code></pre></div></div>
<p>It’s not clear why a <code class="highlighter-rouge">Proxy</code> requires checked exceptions to be declared when normal methods do not. Libraries providing <code class="highlighter-rouge">suspend fun</code> support through a <code class="highlighter-rouge">Proxy</code> will need to be mindful of this behavior and put similar workarounds in place.</p>
<p>This bug fix is available in Retrofit 2.6.1 today!</p>
R8 Optimization: Method Outlining2019-04-11T00:00:00+00:00https://jakewharton.com/r8-optimization-method-outlining<blockquote>
<p>Note: This post is part of a series on D8 and R8, Android’s new dexer and optimizer, respectively. For an intro to D8 read <a href="/androids-java-8-support/">“Android’s Java 8 support”</a>. For an intro to R8 read <a href="/r8-optimization-staticization/">“R8 Optimization: Staticization”</a>.</p>
</blockquote>
<p>I recently wrote about <a href="/the-economics-of-generated-code/">the economics of generated code</a> which talked about performing optimizations to generated code that aren’t worthwhile in manually-written code. While the examples in that post were motivated by changes to code generators that I had worked on in the past, it also resulted in some new changes being made.</p>
<p>One change proposed to <a href="https://github.com/square/moshi/">Moshi</a>, a JSON serializer, replaced its generated strings with <code class="highlighter-rouge">StringBuilder</code> to de-duplicate the constant parts. Each non-null property in your JSON model generates an exception to ensure non-null values are read from the JSON.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> name = stringAdapter.fromJson(reader) ?:
throw JsonDataException(
<span class="gd">- "Non-null value 'name' was null at ${reader.path}")
</span><span class="gi">+ StringBuilder("Non-null value '").append("name")
+ .append("' was null at ").append(reader.path).toString())
</span></code></pre></div></div>
<p>A second exception is generated when that non-null property lacks a default value and no value was present in the JSON.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> return Person(
name = name ?: throw JsonDataException(
<span class="gd">- "Required property 'name' missing at ${reader.path}"),
</span><span class="gi">+ StringBuilder("Required property '").append("name")
+ .append("' missing at ").append(reader.path).toString()),
</span></code></pre></div></div>
<p>These two diffs are the result of applying the advice from that post.</p>
<p>Each of these exceptions are generated for every property in the type. This means if you have a type with 10 properties you get 20 exceptions generated (assuming they’re non-null and don’t have default values). This winds up creating a lot of <code class="highlighter-rouge">StringBuilder</code> bytecode!</p>
<p>One way to reduce this bytecode bloat is to generate a private method which takes four arguments (prefix, name, suffix, path) and returns the final string. This was proposed as part of the change to Moshi. We ultimately duplicated the code instead of generating a method because it ends up optimizing to a smaller APK thanks to R8. Let’s find out why.</p>
<h3 id="representative-example">Representative Example</h3>
<p>Instead of dealing with Moshi, kapt, and generated Kotlin directly, it’s easier to work with a representative example. To start with, we need some JSON model objects. In order to require both of the <code class="highlighter-rouge">StringBuilder</code> usages from above, each property has a non-null type and has no default value.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">data class</span> <span class="nc">User</span><span class="p">(</span>
<span class="kd">val</span> <span class="py">id</span><span class="p">:</span> <span class="nc">String</span><span class="p">,</span>
<span class="kd">val</span> <span class="py">username</span><span class="p">:</span> <span class="nc">String</span><span class="p">,</span>
<span class="kd">val</span> <span class="py">displayName</span><span class="p">:</span> <span class="nc">String</span><span class="p">,</span>
<span class="kd">val</span> <span class="py">email</span><span class="p">:</span> <span class="nc">String</span><span class="p">,</span>
<span class="kd">val</span> <span class="py">created</span><span class="p">:</span> <span class="nc">OffsetDateTime</span><span class="p">,</span>
<span class="kd">val</span> <span class="py">isPublic</span><span class="p">:</span> <span class="nc">Boolean</span>
<span class="p">)</span>
<span class="kd">data class</span> <span class="nc">Tweet</span><span class="p">(</span>
<span class="kd">val</span> <span class="py">id</span><span class="p">:</span> <span class="nc">String</span><span class="p">,</span>
<span class="kd">val</span> <span class="py">userId</span><span class="p">:</span> <span class="nc">String</span><span class="p">,</span>
<span class="kd">val</span> <span class="py">content</span><span class="p">:</span> <span class="nc">String</span><span class="p">,</span>
<span class="kd">val</span> <span class="py">created</span><span class="p">:</span> <span class="nc">OffsetDateTime</span>
<span class="p">)</span>
</code></pre></div></div>
<p>When used with Moshi, these types would be annotated with <code class="highlighter-rouge">@JsonClass</code> which causes the annotation processor to generate code. That code then interacts with Moshi’s <code class="highlighter-rouge">JsonReader</code> type to parse the values of each property. We can replicate this using Android’s built-in <code class="highlighter-rouge">JsonReader</code> type and writing the generated code by hand.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">object</span> <span class="nc">TweetParser</span> <span class="p">{</span>
<span class="k">fun</span> <span class="nf">fromJson</span><span class="p">(</span><span class="n">reader</span><span class="p">:</span> <span class="nc">JsonReader</span><span class="p">):</span> <span class="nc">Tweet</span> <span class="p">{</span>
<span class="kd">var</span> <span class="py">id</span><span class="p">:</span> <span class="nc">String</span><span class="p">?</span> <span class="p">=</span> <span class="k">null</span>
<span class="c1">// other properties…</span>
<span class="n">reader</span><span class="p">.</span><span class="nf">beginObject</span><span class="p">()</span>
<span class="k">while</span> <span class="p">(</span><span class="n">reader</span><span class="p">.</span><span class="nf">peek</span><span class="p">()</span> <span class="p">!=</span> <span class="nc">JsonToken</span><span class="p">.</span><span class="nc">END_OBJECT</span><span class="p">)</span> <span class="p">{</span>
<span class="k">when</span> <span class="p">(</span><span class="n">reader</span><span class="p">.</span><span class="nf">nextName</span><span class="p">())</span> <span class="p">{</span>
<span class="s">"id"</span> <span class="p">-></span> <span class="n">id</span> <span class="p">=</span> <span class="n">reader</span><span class="p">.</span><span class="nf">nextString</span><span class="p">()</span> <span class="o">?:</span>
<span class="k">throw</span> <span class="nc">IllegalStateException</span><span class="p">(</span>
<span class="nc">StringBuilder</span><span class="p">(</span><span class="s">"Non-null value '"</span><span class="p">).</span><span class="nf">append</span><span class="p">(</span><span class="s">"id"</span><span class="p">)</span>
<span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="s">"' was null at"</span><span class="p">).</span><span class="nf">append</span><span class="p">(</span><span class="n">reader</span><span class="p">).</span><span class="nf">toString</span><span class="p">())</span>
<span class="c1">// other properties…</span>
<span class="k">else</span> <span class="p">-></span> <span class="n">reader</span><span class="p">.</span><span class="nf">skipValue</span><span class="p">()</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">reader</span><span class="p">.</span><span class="nf">endObject</span><span class="p">()</span>
<span class="k">return</span> <span class="nc">Tweet</span><span class="p">(</span>
<span class="n">id</span> <span class="p">=</span> <span class="n">id</span> <span class="o">?:</span> <span class="k">throw</span> <span class="nc">IllegalStateException</span><span class="p">(</span>
<span class="nc">StringBuilder</span><span class="p">(</span><span class="s">"Required property '"</span><span class="p">).</span><span class="nf">append</span><span class="p">(</span><span class="s">"id"</span><span class="p">)</span>
<span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="s">"' missing at "</span><span class="p">).</span><span class="nf">append</span><span class="p">(</span><span class="n">reader</span><span class="p">).</span><span class="nf">toString</span><span class="p">()),</span>
<span class="c1">// other properties…</span>
<span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This is the version for <code class="highlighter-rouge">Tweet</code> showing only one property. You would do the same for the <code class="highlighter-rouge">userId</code>, <code class="highlighter-rouge">content</code> and <code class="highlighter-rouge">created</code> properties, and create a similar type for parsing <code class="highlighter-rouge">User</code><sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup>.</p>
<p>If you compile with <code class="highlighter-rouge">kotlinc</code>, dex with D8, and dump the bytecode with <code class="highlighter-rouge">dexdump</code> you’ll see the <code class="highlighter-rouge">StringBuilder</code> code repeated many times.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>0181: new-instance v1, Ljava/lang/IllegalStateException;
0183: new-instance v4, Ljava/lang/StringBuilder;
0185: invoke-direct {v4, v3}, Ljava/lang/StringBuilder;.<init>:(Ljava/lang/String;)V
0188: invoke-virtual {v4, v12}, Ljava/lang/StringBuilder;.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
018b: invoke-virtual {v4, v2}, Ljava/lang/StringBuilder;.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
018e: invoke-virtual {v4, v0}, Ljava/lang/StringBuilder;.append:(Ljava/lang/Object;)Ljava/lang/StringBuilder;
0191: invoke-virtual {v4}, Ljava/lang/StringBuilder;.toString:()Ljava/lang/String;
0194: move-result-object v0
0195: invoke-direct {v1, v0}, Ljava/lang/IllegalStateException;.<init>:(Ljava/lang/String;)V
0198: check-cast v1, Ljava/lang/Throwable;
019a: throw v1
</code></pre></div></div>
<p>This bytecode sequence weighs less than generating the single string so its a net win no matter what, but this still feels like a waste. Generating a method with this code in each parser type would reduce its impact. So why did we elect not to?</p>
<h3 id="outlining">Outlining</h3>
<p>Most of the posts in this R8 series have touched on <em>inlining</em> in one way or another. This optimization is when a method is small enough and/or called infrequently enough that it becomes beneficial to copy the method body contents to the call site and remove the method. <em>Outlining</em> is the opposite optimization where common bytecode sequences are identified and extracted to a shared method.</p>
<p>Before running R8, let’s add a <code class="highlighter-rouge">main</code> function which uses our parsers and can serve as an entry point for optimization.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fun</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span>
<span class="nf">println</span><span class="p">(</span><span class="nc">TweetParser</span><span class="p">.</span><span class="nf">fromJson</span><span class="p">(</span><span class="nc">JsonReader</span><span class="p">(</span><span class="nc">StringReader</span><span class="p">(</span><span class="s">""</span><span class="p">))))</span>
<span class="nf">println</span><span class="p">(</span><span class="nc">UserParser</span><span class="p">.</span><span class="nf">fromJson</span><span class="p">(</span><span class="nc">JsonReader</span><span class="p">(</span><span class="nc">StringReader</span><span class="p">(</span><span class="s">""</span><span class="p">))))</span>
<span class="p">}</span>
</code></pre></div></div>
<p>We don’t need real data because we’re not executing the code. This is just enough to ensure R8 keeps the codepaths we care about. With some simple rules to keep the <code class="highlighter-rouge">main</code> method, let’s see what R8 does.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cat rules.txt
-keepclasseswithmembers class * {
public static void main(...);
}
-dontobfuscate
$ java -jar r8.jar \
--lib $ANDROID_HOME/platforms/android-28/android.jar \
--release \
--output . \
--pg-conf rules.txt \
*.class
</code></pre></div></div>
<p>Dumping the output of R8 shows a very different picture for the exception code compared to what D8 produced.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> 0181: new-instance v1, Ljava/lang/IllegalStateException;
<span class="gd">-0183: new-instance v4, Ljava/lang/StringBuilder;
-0185: invoke-direct {v4, v3}, Ljava/lang/StringBuilder;.<init>:(Ljava/lang/String;)V
-0188: invoke-virtual {v4, v12}, Ljava/lang/StringBuilder;.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
-018b: invoke-virtual {v4, v2}, Ljava/lang/StringBuilder;.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
-018e: invoke-virtual {v4, v0}, Ljava/lang/StringBuilder;.append:(Ljava/lang/Object;)Ljava/lang/StringBuilder;
-0191: invoke-virtual {v4}, Ljava/lang/StringBuilder;.toString:()Ljava/lang/String;
</span><span class="gi">+0183: invoke-static {v3, v12, v2, v0}, Lcom/android/tools/r8/GeneratedOutlineSupport;.outline0:(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/Object;)Ljava/lang/String;
</span> 0194: move-result-object v0
0195: invoke-direct {v1, v0}, Ljava/lang/IllegalStateException;.<init>:(Ljava/lang/String;)V
<span class="gd">-0198: check-cast v1, Ljava/lang/Throwable;
</span> 019a: throw v1
</code></pre></div></div>
<p>The outlining optimization has recognized that the <code class="highlighter-rouge">StringBuilder</code> code is repeated many times. The bytecode sequence is de-duplicated to the <code class="highlighter-rouge">outline0</code> method on this <code class="highlighter-rouge">com.android.tools.r8.GeneratedOutlineSupport</code> class. Every occurrence of the bytecode sequence is replaced with a call to this new method.</p>
<p>Taking a look at the new method shows the common <code class="highlighter-rouge">StringBuilder</code> code.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[000eb4] com.android.tools.r8.GeneratedOutlineSupport.outline0:(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/Object;)Ljava/lang/String;
0000: new-instance v0, Ljava/lang/StringBuilder
0002: invoke-direct {v0, v1}, Ljava/lang/StringBuilder;.<init>:(Ljava/lang/String;)V
0005: invoke-virtual {v0, v2}, Ljava/lang/StringBuilder;.append:(Ljava/lang/String;)Ljava/lang/StringBuilder
0008: invoke-virtual {v0, v3}, Ljava/lang/StringBuilder;.append:(Ljava/lang/String;)Ljava/lang/StringBuilder
000b: invoke-virtual {v0, v4}, Ljava/lang/StringBuilder;.append:(Ljava/lang/Object;)Ljava/lang/StringBuilder
000e: invoke-virtual {v0}, Ljava/lang/StringBuilder;.toString:()Ljava/lang/String
0011: move-result-object v1
0012: return-object v1
</code></pre></div></div>
<p>R8 has created the helper method which we were considering adding ourselves!</p>
<p>I specifically chose to use two types in the example which together have 10 properties resulting in 20 <code class="highlighter-rouge">StringBuilder</code> usages. This is the lower bound of duplicate sequences that R8 will consider outlining. The duplicated bytecode must also be between 3 and 99 bytes.</p>
<p>If Moshi generated a private <code class="highlighter-rouge">StringBuidler</code> helper method our example would still have two copies. You would need 20 JSON model objects before R8 stepped in and de-duplicated the helper method. By electing to duplicate the <code class="highlighter-rouge">StringBuilder</code> code, only 20 <em>properties</em> are needed in any number of JSON model objects before R8 outlining kicks in. Once that happens we only pay for the code once no matter how many JSON model objects and properties are in use.</p>
<hr />
<p>Outlining works really well with generated code since it tends to produce repeated patterns. In examples like the one above, you can avoid putting a helper function in your runtime library and instead rely on R8 to de-duplicate bytecode when it’s repeated enough. And because R8 is doing whole-program analysis, unrelated code which happens to have the same bytecode patterns participate in the de-duplication.</p>
<p>It’s also interesting to think about how this interacts with Kotlin’s <code class="highlighter-rouge">inline</code> function modifier. The more you use inline functions (and especially if you invoke inline functions inside other inline functions) the more likely you are to have R8 outline some of the function body back into a regular method. Make sure that you’re using <code class="highlighter-rouge">inline</code> for things like <code class="highlighter-rouge">reified</code> generics or to avoid allocating lambda objects as it’s intended.</p>
<p>In the <a href="/r8-optimization-class-constant-operations/">previous post about R8</a> I teased that the next post (aka this one) would cover an optimization that created <code class="highlighter-rouge">const-class</code> bytecodes. After writing two posts outside of this series on generated code and having the discussion on the Moshi change, however, it felt like a natural progression to cover outlining. With outlining out of the way the next R8 post will get back on track with producing <code class="highlighter-rouge">const-class</code> bytecodes.</p>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>The full example code is available at <a href="https://gist.github.com/JakeWharton/6d08b7fb74c320b048db68e21912d878">gist.github.com/JakeWharton/6d08b7fb74c320b048db68e21912d878</a> <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
Optimizing Bytecode by Manipulating Source Code2019-04-02T00:00:00+00:00https://jakewharton.com/optimizing-bytecode-by-manipulating-source-code<p>This post is a follow-up to “<a href="/the-economics-of-generated-code/">The Economics of Generated Code</a>” which argued that spending time optimizing generated code is more worthwhile than the same optimizations done in manually-written code.</p>
<p>The second example from that post dealt with looking up views, checking for null, and potentially throwing an exception. In an effort to reduce the impact of the generated exception message string, each was split into a prefix which will be de-duplicated and the view ID name which was effectively free since it matched a field name. If you’re lost on what that all means, check out the other post first.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> public static MainBinding bind(View root) {
TextView name = root.findViewById(R.id.name);
if (name == null) {
<span class="gd">- throw new NullPointerException("View 'name' required but not found");
</span><span class="gi">+ throw new NullPointerException("Missing required view with ID: ".concat("name"));
</span> }
TextView email = root.findViewById(R.id.email);
if (email == null) {
<span class="gd">- throw new NullPointerException("View 'email' required but not found");
</span><span class="gi">+ throw new NullPointerException("Missing required view with ID: ".concat("email"));
</span> }
return new MainBinding(root, name, email);
}
</code></pre></div></div>
<p>That change was just about strings, but I also mentioned that there’s more optimization which could be done. So let’s do it!</p>
<p>By virtue of the fact that we throw an exception when a view is absent, that case is expected to be rare. This is what allowed us to justify sacrificing a single string constant in favor of multiple constants and runtime concatenation. While that allowed us to de-duplicate the strings, it creates more duplication in the bytecode.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[000288] MainBinding.bind:(Landroid/view/View;)LMainBinding;
0000: sget v0, LR$id;.name:I
0002: invoke-virtual {v3, v0}, Landroid/view/View;.findViewById:(I)Landroid/view/View;
0005: move-result-object v0
0006: check-cast v0, Landroid/widget/TextView;
0008: if-nez v0, 0018
000a: new-instance v0, Ljava/lang/NullPointerException;
000c: const-string v1, "Missing required view with ID: "
000e: const-string v2, "name"
0010: invoke-virtual {v1, v2}, Ljava/lang/String;.concat:(Ljava/lang/String;)Ljava/lang/String;
0013: move-result-object v1
0014: invoke-direct {v0, v1}, Ljava/lang/NullPointerException;.<init>:(Ljava/lang/String;)V
0017: throw v0
0018: sget v1, LR$id;.email:I
001a: invoke-virtual {v3, v1}, Landroid/view/View;.findViewById:(I)Landroid/view/View;
001d: move-result-object v1
001e: check-cast v1, Landroid/widget/TextView;
0020: if-nez v1, 0030
0022: new-instance v0, Ljava/lang/NullPointerException;
0024: const-string v1, "Missing required view with ID: "
0026: const-string v2, "email"
0028: invoke-virtual {v1, v2}, Ljava/lang/String;.concat:(Ljava/lang/String;)Ljava/lang/String;
002b: move-result-object v1
002c: invoke-direct {v0, v1}, Ljava/lang/NullPointerException;.<init>:(Ljava/lang/String;)V
002f: throw v0
0030: new-instance v2, LMainBinding;
0032: invoke-direct {v2, v3, v0, v1}, LMainBinding;.<init>:(Landroid/view/View;Landroid/widget/TextView;Landroid/widget/TextView;)V
0035: return-object v2
</code></pre></div></div>
<p>I’ve spaced the bytecode out so it’s easier to see the logical sections and, hopefully, identify what we want to change.</p>
<p>Indices <code class="highlighter-rouge">000a</code>–<code class="highlighter-rouge">0017</code> and <code class="highlighter-rouge">0022</code>–<code class="highlighter-rouge">002f</code> are near-exact duplicates of each other which only vary by the name of the missing view. Again, because this code is expected to never run, it would be nice to remove the duplication. Fixing this will be the focus of the post, but I also want to point out a second problem that we’ll fix in tandem.</p>
<p>In addition to the exception code being duplicated it’s also interspersed between “normal” code. This means that the common execution path of required views being present has to jump over unused bytecode.</p>
<p><img src="/static/post-image/bytecode-economics-1.png" alt="" />
<!--
digraph {
ranksep=.2;
nodesep=0;
node [fontsize=10, height=.05, fontname="menlo"];
{
node [shape=plaintext];
edge [style=invis];
"0000" -> "0008" -> "000a" -> "0018" -> "0020" -> "0022" -> "0030";
}
{
node [shape=plaintext, width=2];
edge [style=invis];
"findViewById(name)" -> "if name == null" -> "throw NPE(name)" -> "findViewById(email)" -> "if email == null" -> "throw NPE(email)" -> "return MainBinding(…)"
}
{
edge [arrowsize=0.5];
"findViewById(name)" -> "if name == null"[constraint=false];
"if name == null" -> "throw NPE(name)"[constraint=false];
"if name == null" -> "findViewById(email)"[constraint=false, margin="4,4"];
"findViewById(email)" -> "if email == null"[constraint=false];
"if email == null" -> "throw NPE(email)"[constraint=false];
"if email == null" -> "return MainBinding(…)"[constraint=false];
}
{ rank=same; "0000"; "findViewById(name)"; }
{ rank=same; "0008"; "if name == null"; }
{ rank=same; "000a"; "throw NPE(name)"; }
{ rank=same; "0018"; "findViewById(email)"; }
{ rank=same; "0020"; "if email == null"; }
{ rank=same; "0022"; "throw NPE(email)"; }
{ rank=same; "0030"; "return MainBinding(…)"; }
}
--></p>
<p>The code was actually compiled with the old <code class="highlighter-rouge">dx</code> tool to produce the bytecode above. Simply compiling with D8 instead produces a dramatically different arrangement of the control flow.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[000258] MainBinding.bind:(Landroid/view/View;)LMainBinding;
0000: sget v0, LR$id;.name:I
0002: invoke-virtual {v3, v0}, Landroid/view/View;.findViewById:(I)Landroid/view/View;
0005: move-result-object v0
0006: check-cast v0, Landroid/widget/TextView;
0008: const-string v1, "Missing required view with ID: "
000a: if-eqz v0, 0028
000c: sget v2, LR$id;.email:I
000e: invoke-virtual {v3, v2}, Landroid/view/View;.findViewById:(I)Landroid/view/View;
0011: move-result-object v2
0012: check-cast v2, Landroid/widget/TextView;
0014: if-eqz v2, 001c
0016: new-instance v1, LMainBinding;
0018: invoke-direct {v1, v3, v0, v2}, LMainBinding;.<init>:(Landroid/view/View;Landroid/widget/TextView;Landroid/widget/TextView;)V
001b: return-object v1
001c: new-instance v3, Ljava/lang/NullPointerException;
001e: const-string v0, "email"
0020: invoke-virtual {v1, v0}, Ljava/lang/String;.concat:(Ljava/lang/String;)Ljava/lang/String;
0023: move-result-object v0
0024: invoke-direct {v3, v0}, Ljava/lang/NullPointerException;.<init>:(Ljava/lang/String;)V
0027: throw v3
0028: new-instance v3, Ljava/lang/NullPointerException;
002a: const-string v0, "name"
002c: invoke-virtual {v1, v0}, Ljava/lang/String;.concat:(Ljava/lang/String;)Ljava/lang/String;
002f: move-result-object v0
0030: invoke-direct {v3, v0}, Ljava/lang/NullPointerException;.<init>:(Ljava/lang/String;)V
0033: throw v3
</code></pre></div></div>
<p>D8 understands that the case in which you throw an exception is, well, exceptional. Thus, the conditionals are inverted so that the exceptional cases move to the end of the method. This makes the common case not require any jumps.</p>
<p><img src="/static/post-image/bytecode-economics-2.png" alt="" />
<!--
digraph {
ranksep=.2;
nodesep=0;
node [fontsize=10, height=.05, fontname="menlo"];
{
node [shape=plaintext];
edge [style=invis];
"0000" -> "0008" -> "000a" -> "000c" -> "0014" -> "0016" -> "001c" -> "0028";
}
{
node [shape=plaintext, width=2];
edge [style=invis];
"findViewById(name)" -> "const-string \"Missing\"" -> "if name != null" -> "findViewById(email)" -> "if email != null" -> "return MainBinding(…)" -> "throw NPE(name)" -> "throw NPE(email)"
}
{
edge [arrowsize=0.5];
"findViewById(name)" -> "const-string \"Missing\""[constraint=false];
"const-string \"Missing\"" -> "if name != null"[constraint=false];
"if name != null" -> "throw NPE(name)"[constraint=false];
"if name != null" -> "findViewById(email)"[constraint=false, margin="4,4"];
"findViewById(email)" -> "if email != null"[constraint=false];
"if email != null" -> "throw NPE(email)"[constraint=false];
"if email != null" -> "return MainBinding(…)"[constraint=false];
}
{ rank=same; "0000"; "findViewById(name)"; }
{ rank=same; "0008"; "const-string \"Missing\""; }
{ rank=same; "000a"; "if name != null"; }
{ rank=same; "000c"; "findViewById(email)"; }
{ rank=same; "0014"; "if email != null"; }
{ rank=same; "0016"; "return MainBinding(…)"; }
{ rank=same; "001c"; "throw NPE(name)"; }
{ rank=same; "0028"; "throw NPE(email)"; }
}
--></p>
<p>Another side-effect of using D8 is that the loading of the exception message prefix string was de-duplicated at bytecode index <code class="highlighter-rouge">0008</code>. This is actually an unfortunate behavior since it now occurs during normal execution as well.</p>
<p>Before attempting to fix these problems, let’s manually re-arrange the bytecode (with dummy indices, for simplicity) to the ideal form we’d like to produce.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[000258] MainBinding.bind:(Landroid/view/View;)LMainBinding;
0000: sget v0, LR$id;.name:I
0001: invoke-virtual {v3, v0}, Landroid/view/View;.findViewById:(I)Landroid/view/View;
0002: move-result-object v0
0003: check-cast v0, Landroid/widget/TextView;
0010: if-eqz v0, 0050
0020: sget v1, LR$id;.email:I
0021: invoke-virtual {v3, v1}, Landroid/view/View;.findViewById:(I)Landroid/view/View;
0022: move-result-object v1
0023: check-cast v1, Landroid/widget/TextView;
0030: if-eqz v1, 0060
0040: new-instance v2, LMainBinding;
0041: invoke-direct {v2, v3, v0, v1}, LMainBinding;.<init>:(Landroid/view/View;Landroid/widget/TextView;Landroid/widget/TextView;)V
0042: return-object v2
0050: const-string v2, "email"
0051: goto 0070
0060: const-string v2, "name"
0070: const-string v1, "Missing required view with ID: "
0071: new-instance v3, Ljava/lang/NullPointerException;
0072: invoke-virtual {v1, v2}, Ljava/lang/String;.concat:(Ljava/lang/String;)Ljava/lang/String;
0073: move-result-object v2
0074: invoke-direct {v3, v2}, Ljava/lang/NullPointerException;.<init>:(Ljava/lang/String;)V
0075: throw v3
</code></pre></div></div>
<p>This has everything we want: the normal execution case flows from index <code class="highlighter-rouge">0000</code> to <code class="highlighter-rouge">0042</code> without jumps and the exception-handling code is de-deuplicated at index <code class="highlighter-rouge">0070</code> to <code class="highlighter-rouge">0075</code>. There’s only one load of the prefix string as part of creating the exception message. When a null is found, the code jumps to a section which loads the correct view ID string and then jumps (or falls through) to the exception.</p>
<p><img src="/static/post-image/bytecode-economics-3.png" alt="" />
<!--
digraph {
ranksep=.2;
nodesep=0;
node [fontsize=10, height=.05, fontname="menlo"];
{
node [shape=plaintext];
edge [style=invis];
"0000" -> "0010" -> "0020" -> "0030" -> "0040" -> "0050" -> "0060" -> "0070";
}
{
node [shape=plaintext, width=2];
edge [style=invis];
"findViewById(name)" -> "if name != null" -> "findViewById(email)" -> "if email != null" -> "return MainBinding(…)" -> "const-string \"name\"" -> "const-string \"email\"" -> "throw NPE(missingId)"
}
{
edge [arrowsize=0.5];
"findViewById(name)" -> "if name != null"[constraint=false];
"if name != null" -> "const-string \"name\""[constraint=false];
"if name != null" -> "findViewById(email)"[constraint=false, margin="4,4"];
"findViewById(email)" -> "if email != null"[constraint=false];
"if email != null" -> "const-string \"email\""[constraint=false];
"if email != null" -> "return MainBinding(…)"[constraint=false];
"const-string \"name\"" -> "throw NPE(missingId)"[constraint=false];
"const-string \"email\"" -> "throw NPE(missingId)"[constraint=false];
}
{ rank=same; "0000"; "findViewById(name)"; }
{ rank=same; "0010"; "if name != null"; }
{ rank=same; "0020"; "findViewById(email)"; }
{ rank=same; "0030"; "if email != null"; }
{ rank=same; "0040"; "return MainBinding(…)"; }
{ rank=same; "0050"; "const-string \"name\""; }
{ rank=same; "0060"; "const-string \"email\""; }
{ rank=same; "0070"; "throw NPE(missingId)"; }
}
--></p>
<p>Now that we have a goal it’s easier to iterate on the generated Java code to see how our changes move us closer or farther from achieving it. Let’s start by de-duplicating the exception code.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> public static MainBinding bind(View root) {
<span class="gi">+ String missingId = null;
</span> TextView name = root.findViewById(R.id.name);
if (name == null) {
<span class="gi">+ missingId = "name";
</span><span class="gd">- throw new NullPointerException("Missing required view with ID: ".concat("name"));
</span> }
TextView email = root.findViewById(R.id.email);
if (email == null) {
<span class="gi">+ missingId = "email";
</span><span class="gd">- throw new NullPointerException("Missing required view with ID: ".concat("email"));
</span> }
<span class="gd">- return new MainBinding(root, name, email);
</span><span class="gi">+ if (missingId == null) {
+ return new MainBinding(root, name, email);
+ }
+ throw new NullPointerException("Missing required view with ID: ".concat(missingId));
</span> }
</code></pre></div></div>
<p>This produces bytecode which successfully de-duplicates the exception code but with a slight penalty on the other parts.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[000258] MainBinding.bind:(Landroid/view/View;)LMainBinding;
0000: sget v0, LR$id;.name:I
0002: invoke-virtual {v3, v0}, Landroid/view/View;.findViewById:(I)Landroid/view/View;
0005: move-result-object v0
0006: check-cast v0, Landroid/widget/TextView;
0008: if-nez v0, 000d
000a: const-string v1, "name"
000c: goto 000e
000d: const/4 v1, #int 0
000e: sget v2, LR$id;.email:I
0010: invoke-virtual {v3, v2}, Landroid/view/View;.findViewById:(I)Landroid/view/View;
0013: move-result-object v2
0014: check-cast v2, Landroid/widget/TextView;
0016: if-nez v2, 001a
0018: const-string v1, "email"
001a: if-nez v1, 0022
001c: new-instance v1, LMainBinding;
001e: invoke-direct {v1, v3, v0, v2}, LMainBinding;.<init>:(Landroid/view/View;Landroid/widget/TextView;Landroid/widget/TextView;)V
0021: return-object v1
0022: new-instance v3, Ljava/lang/NullPointerException;
0024: const-string v0, "Missing required view with ID: "
0026: invoke-virtual {v0, v1}, Ljava/lang/String;.concat:(Ljava/lang/String;)Ljava/lang/String;
0029: move-result-object v0
002a: invoke-direct {v3, v0}, Ljava/lang/NullPointerException;.<init>:(Ljava/lang/String;)V
002d: throw v3
</code></pre></div></div>
<p>Since the <code class="highlighter-rouge">throw</code> statement was removed from the <code class="highlighter-rouge">if</code> check body, D8 no longer understands that they’re exceptional cases. This means that the jumps in normal execution have returned. There’s also a slight behavior change in that we now report the last missing view instead of the first.</p>
<p><img src="/static/post-image/bytecode-economics-4.png" alt="" />
<!--
digraph {
ranksep=.2;
nodesep=0;
node [fontsize=10, height=.05, fontname="menlo"];
{
node [shape=plaintext];
edge [style=invis];
"0000" -> "0008" -> "000a" -> "000d" -> "000e" -> "0016" -> "0018" -> "001a" -> "001c" -> "0022";
}
{
node [shape=plaintext, width=2];
edge [style=invis];
"findViewById(name)" -> "if name == null" -> "missingId = \"name\"" -> "missingId = null" -> "findViewById(email)" -> "if email == null" -> "missingId = \"email\"" -> "if missingId == null" -> "return MainBinding(…)" -> "throw NPE(missingId)"
}
{
edge [arrowsize=0.5];
"findViewById(name)" -> "if name == null"[constraint=false];
"if name == null" -> "missingId = \"name\""[constraint=false];
"if name == null" -> "missingId = null"[constraint=false, margin="4,4"];
"missingId = \"name\"" -> "findViewById(email)"[constraint=false];
"missingId = null" -> "findViewById(email)"[constraint=false];
"findViewById(email)" -> "if email == null"[constraint=false];
"if email == null" -> "missingId = \"email\""[constraint=false];
"if email == null" -> "if missingId == null"[constraint=false];
"missingId = \"email\"" -> "if missingId == null"[constraint=false];
"if missingId == null" -> "return MainBinding(…)"[constraint=false];
"if missingId == null" -> "throw NPE(missingId)"[constraint=false];
}
{ rank=same; "0000"; "findViewById(name)"; }
{ rank=same; "0008"; "if name == null"; }
{ rank=same; "000a"; "missingId = \"name\""; }
{ rank=same; "000d"; "missingId = null"; }
{ rank=same; "000e"; "findViewById(email)"; }
{ rank=same; "0016"; "if email == null"; }
{ rank=same; "0018"; "missingId = \"email\""; }
{ rank=same; "001a"; "if missingId == null"; }
{ rank=same; "001c"; "return MainBinding(…)"; }
{ rank=same; "0022"; "throw NPE(missingId)"; }
}
--></p>
<p>The first thing that comes to my mind for trying to eliminate the needless jumps is nesting the conditionals.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> public static MainBinding bind(View root) {
<span class="gd">- String missingId = null;
</span><span class="gi">+ String missingId;
</span> TextView name = root.findViewById(R.id.name);
<span class="gd">- if (name == null) {
- missingId = "name";
- }
- TextView email = root.findViewById(R.id.email);
- if (email == null) {
- missingId = "email";
- }
- if (missingId == null) {
- return new MainBinding(root, name, email);
</span><span class="gi">+ if (name != null) {
+ TextView email = root.findViewById(R.id.email);
+ if (email != null) {
+ return new MainBinding(root, name, email);
+ } else {
+ missingId = "email";
+ }
+ } else {
+ missingId = "name";
</span> }
throw new NullPointerException("Missing required view with ID: ".concat(missingId));
}
</code></pre></div></div>
<p>Lo and behold, we’ve done it!</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[000258] MainBinding.bind:(Landroid/view/View;)LMainBinding;
0000: sget v0, LR$id;.name:I
0002: invoke-virtual {v3, v0}, Landroid/view/View;.findViewById:(I)Landroid/view/View;
0005: move-result-object v0
0006: check-cast v0, Landroid/widget/TextView;
0008: if-eqz v0, 001d
000a: sget v1, LR$id;.email:I
000c: invoke-virtual {v3, v1}, Landroid/view/View;.findViewById:(I)Landroid/view/View;
000f: move-result-object v1
0010: check-cast v1, Landroid/widget/TextView;
0012: if-eqz v1, 001a
0014: new-instance v2, LMainBinding;
0016: invoke-direct {v2, v3, v0, v1}, LMainBinding;.<init>:(Landroid/view/View;Landroid/widget/TextView;Landroid/widget/TextView;)V
0019: return-object v2
001a: const-string v3, "email"
001c: goto 001f
001d: const-string v3, "name"
001f: new-instance v0, Ljava/lang/NullPointerException;
0021: const-string v1, "Missing required view with ID: "
0023: invoke-virtual {v1, v3}, Ljava/lang/String;.concat:(Ljava/lang/String;)Ljava/lang/String;
0026: move-result-object v3
0027: invoke-direct {v0, v3}, Ljava/lang/NullPointerException;.<init>:(Ljava/lang/String;)V
002a: throw v0
</code></pre></div></div>
<p>Modulo a few register re-numberings, this is <em>exactly</em> the same bytecode as the ideal case we crafted above. The key which makes this work is mostly in the <code class="highlighter-rouge">else</code> branches. Once an <code class="highlighter-rouge">else</code> branch is taken, it then immediately jumps down to the exception code because it’s the last statement in the <code class="highlighter-rouge">if</code> branch in every layer above.</p>
<p><img src="/static/post-image/bytecode-economics-5.png" alt="" />
<!--
digraph {
ranksep=.2;
nodesep=0;
node [fontsize=10, height=.05, fontname="menlo"];
{
node [shape=plaintext];
edge [style=invis];
"0000" -> "0008" -> "000a" -> "0012" -> "0014" -> "001a" -> "001d" -> "001f";
}
{
node [shape=plaintext, width=2];
edge [style=invis];
"findViewById(name)" -> "if name != null" -> "findViewById(email)" -> "if email != null" -> "return MainBinding(…)" -> "const-string \"name\"" -> "const-string \"email\"" -> "throw NPE(missingId)"
}
{
edge [arrowsize=0.5];
"findViewById(name)" -> "if name != null"[constraint=false];
"if name != null" -> "const-string \"name\""[constraint=false];
"if name != null" -> "findViewById(email)"[constraint=false, margin="4,4"];
"findViewById(email)" -> "if email != null"[constraint=false];
"if email != null" -> "const-string \"email\""[constraint=false];
"if email != null" -> "return MainBinding(…)"[constraint=false];
"const-string \"name\"" -> "throw NPE(missingId)"[constraint=false];
"const-string \"email\"" -> "throw NPE(missingId)"[constraint=false];
}
{ rank=same; "0000"; "findViewById(name)"; }
{ rank=same; "0008"; "if name != null"; }
{ rank=same; "000a"; "findViewById(email)"; }
{ rank=same; "0012"; "if email != null"; }
{ rank=same; "0014"; "return MainBinding(…)"; }
{ rank=same; "001a"; "const-string \"name\""; }
{ rank=same; "001d"; "const-string \"email\""; }
{ rank=same; "001f"; "throw NPE(missingId)"; }
}
--></p>
<p>So are we done?</p>
<p>While we shouldn’t care too much about how generated code looks, I still find this solution to be unsatisfactory. If you have 20 views in a layout you’ll get 20 levels of nesting. Even though generated code isn’t written by hand, you still might find yourself reading it when clicking through elements of a stacktrace or during debugging. As a result, if a more readable solution is available without sacrificing the value we should prefer it.</p>
<p>In order to flatten the generated code, we need a similar mechanism which allows control flow to jump to a particular point. This sounds awfully similar to a “goto”, and it is, but <em>all</em> control flow is a form of “goto” so we might as well use whatever the language provides. For Java, the <code class="highlighter-rouge">break</code> statement of a <code class="highlighter-rouge">switch</code> or loop comes to mind as something to try.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> public static MainBinding bind(View root) {
String missingId;
<span class="gd">- TextView name = root.findViewById(R.id.name);
- if (name != null) {
</span><span class="gi">+ while (true) {
+ TextView name = root.findViewById(R.id.name);
+ if (name == null) {
+ missingId = "name";
+ break;
+ }
</span> TextView email = root.findViewById(R.id.email);
<span class="gd">- if (email != null) {
- return new MainBinding(root, name, email);
- } else {
</span><span class="gi">+ if (email == null) {
</span> missingId = "email";
<span class="gi">+ break;
</span> }
<span class="gd">- } else {
- missingId = "name";
</span><span class="gi">+ return new MainBinding(root, name, email);
</span> }
throw new NullPointerException("Missing required view with ID: ".concat(missingId));
}
</code></pre></div></div>
<p>By using a <code class="highlighter-rouge">return</code> statement as the last of the infinite loop, we never actually loop and instead just borrow the <code class="highlighter-rouge">break</code> feature. This is functionality equivalent to the previous version and it produces the exact same bytecode but without nesting.</p>
<p>Does a loop that doesn’t actually loop offend your sensibilities? It certainly does for IntelliJ IDEA which produces a warning: “‘while’ loop does not loop”. We could generate a suppression, but it would be nice to just use something else more suited for this case. There’s actually one more construct where a <code class="highlighter-rouge">break</code> can be used: labeled blocks.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> public static MainBinding bind(View root) {
String missingId;
<span class="gd">- while (true) {
</span><span class="gi">+ missingId: {
</span> TextView name = root.findViewById(R.id.name);
if (name == null) {
missingId = "name";
<span class="gd">- break;
</span><span class="gi">+ break missingId;
</span> }
TextView email = root.findViewById(R.id.email);
if (email == null) {
missingId = "email";
<span class="gd">- break;
</span><span class="gi">+ break missingId;
</span> }
return new MainBinding(root, name, email);
}
throw new NullPointerException("Missing required view with ID: ".concat(missingId));
}
</code></pre></div></div>
<p>Now this <em>really</em> looks like a “goto”, but the compiler will still validate that <code class="highlighter-rouge">missingId</code> is initialized in all execution paths that lead to the exception just like it did with <code class="highlighter-rouge">while (true)</code> and the nested <code class="highlighter-rouge">if</code>/<code class="highlighter-rouge">else</code>s. And, unsurprisingly, the bytecode remains the same.</p>
<p>This is the final form of this specific example of generated code as it stands right now. The bytecode size was reduced from 55 bytes to 31. The duplication was removed and the control flow is now tailored for all views being present. The source code actually got a little bit longer, but it’s still very readable. The labeled block is admittedly something you don’t see often and probably wouldn’t use in manually written code unless it was for breaking across nested loops.</p>
<p>You don’t need to dig this deep if you’re building something that generates code. Start with generating a good API and producing correct behavior. All of this optimization can be done later, or even never. I get involved in this optimization because it’s a fun exploration, but also because <em>the economics of generated code</em> mean that the work almost always pays for itself.</p>
The Economics of Generated Code2019-03-26T00:00:00+00:00https://jakewharton.com/the-economics-of-generated-code<p>Among the many things that I’ve <del>stolen</del> learned from <a href="https://twitter.com/jessewilson">Jesse Wilson</a> is the phrase “the economics of generated code”. This captures the idea that the things we value when generating code are different than those we value for code that’s manually written.</p>
<p>A code generator is only written once but the code it generates occurs many times. Thus, any investment into making the generator emit more efficient code will pay for itself very quickly. This generally means output less code and allocate fewer objects wherever possible. I’d like to expand on that with two specific, real-world examples which I’ve run into.</p>
<h3 id="extra-method-references">Extra Method References</h3>
<p>While it’s not as much of a problem as it used to be, method reference count is still something worth keeping an eye on. This is especially true for generated code. Small changes in the generator can result in the count going up or down by the hundreds or thousands.</p>
<p>It’s common for generated classes to be a subtype of a class in the runtime library. Aside from facilitating polymorphism, this allows consolidating common utilities and behavior. Take a JSON model that wants to retain unknown keys and values encountered during parsing. Each generated class could maintain its own <code class="highlighter-rouge">Map<String, ?></code> for the unknown pairs, but this is a great candidate for consolidation into a base class in the library.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">abstract</span> <span class="kd">class</span> <span class="nc">JsonModel</span> <span class="o">{</span>
<span class="kd">private</span> <span class="kd">final</span> <span class="nc">Map</span><span class="o"><</span><span class="nc">String</span><span class="o">,</span> <span class="o">?></span> <span class="n">unknownPairs</span><span class="o">;</span>
<span class="kd">public</span> <span class="kd">final</span> <span class="nc">Map</span><span class="o"><</span><span class="nc">String</span><span class="o">,</span> <span class="o">?></span> <span class="n">getUnknownPairs</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">unknownPairs</span><span class="o">;</span>
<span class="o">}</span>
<span class="c1">// …</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Not having a <code class="highlighter-rouge">getUnknownPairs()</code> method in each generated class should obviously reduce the count. But since the count is not just about declared methods, reducing the <em>referenced</em> methods in the generated code will also have an impact.</p>
<p>Each generated class extends <code class="highlighter-rouge">JsonModel</code> and implements <code class="highlighter-rouge">toString()</code> which outputs its own fields and the <code class="highlighter-rouge">getUnknownPairs()</code> map.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">final</span> <span class="kd">class</span> <span class="nc">UserModel</span> <span class="kd">extends</span> <span class="nc">JsonModel</span> <span class="o">{</span>
<span class="kd">private</span> <span class="kd">final</span> <span class="nc">String</span> <span class="n">name</span><span class="o">;</span>
<span class="kd">private</span> <span class="kd">final</span> <span class="nc">String</span> <span class="n">email</span><span class="o">;</span>
<span class="c1">// …</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="nc">String</span> <span class="nf">toString</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="s">"UserModel{"</span>
<span class="o">+</span> <span class="s">"name="</span> <span class="o">+</span> <span class="n">name</span> <span class="o">+</span> <span class="s">", "</span>
<span class="o">+</span> <span class="s">"email="</span> <span class="o">+</span> <span class="n">email</span> <span class="o">+</span> <span class="s">", "</span>
<span class="o">+</span> <span class="s">"unknownPairs="</span> <span class="o">+</span> <span class="n">getUnknownPairs</span><span class="o">()</span>
<span class="o">+</span> <span class="sc">'}'</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>When you compile, dex, and dump the Dalvik bytecode of the above class with <code class="highlighter-rouge">dexdump</code>, the way in which <code class="highlighter-rouge">toString()</code> invokes the <code class="highlighter-rouge">getUnknownPairs()</code> method is surprising.</p>
<pre class="highlight">
[00024c] UserModel.toString:()Ljava/lang/String;
0000: iget-object v0, v5, LUserModel;.name:Ljava/lang/String;
0002: iget-object v1, v5, LUserModel;.email:Ljava/lang/String;
<b>0004: invoke-virtual {v5}, LUserModel;.getUnknownPairs:()Ljava/util/Map;</b>
0007: move-result-object v2
</pre>
<p>Despite placing the <code class="highlighter-rouge">getUnknownPairs()</code> method on the <code class="highlighter-rouge">JsonModel</code> supertype, each generated class produces a reference to that method as if it were defined directly on the generated type. Moving the method does not actually reduce the count!</p>
<p>A medium-sized app might have 100 models for its API layer. If each generated class contains four calls to a method defined in the supertype that’s 400 method references created for no purpose.</p>
<p>Changing the generated code to explicitly use <code class="highlighter-rouge">super</code> will produce method references which all point directly to the supertype method.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> @Override public String toString() {
return "UserModel{"
+ "name=" + name + ", "
+ "email=" + email + ", "
<span class="gd">- + "unknownPairs=" + getUnknownPairs()
</span><span class="gi">+ + "unknownPairs=" + super.getUnknownPairs()
</span> + '}';
}
</code></pre></div></div>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> [00024c] UserModel.toString:()Ljava/lang/String;
0000: iget-object v0, v5, LUserModel;.name:Ljava/lang/String;
0002: iget-object v1, v5, LUserModel;.email:Ljava/lang/String;
<span class="gd">-0004: invoke-virtual {v5}, LUserModel;.getUnknownPairs:()Ljava/util/Map;
</span><span class="gi">+0004: invoke-virtual {v5}, LJsonModel;.getUnknownPairs:()Ljava/util/Map;
</span> 0007: move-result-object v2
</code></pre></div></div>
<p>Those 400 extra references are now reduced to just one! We would normally be unlikely to make such a change, but because we control the base class and the generated class this change is safe and results in a significant reduction of method references.</p>
<p>It’s important to point out that using R8 to optimize your app will change this method reference automatically. Not every consumer of your code generator will be using an optimizer, though. Making this small change will ensure everyone benefits.</p>
<h3 id="string-duplication">String Duplication</h3>
<p>Having strings in generated code isn’t a given, but it shows up frequently enough to think about its impact. In my experience, strings in generated code tend to fall into two categories: keys for some type of serialization or error messages for exceptions. There’s not much we can do about the former, but the latter is interesting because those strings exist in code paths which are expected to be rarely taken.</p>
<p>Take, for example, a code generator which binds Android views from a layout into fields of a class. Views are required when they’re present in every configuration of the layout and we validate their presence at runtime with a null check.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">final</span> <span class="kd">class</span> <span class="nc">MainBinding</span> <span class="o">{</span>
<span class="c1">// …</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="nc">MainBinding</span> <span class="nf">bind</span><span class="o">(</span><span class="nc">View</span> <span class="n">root</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">TextView</span> <span class="n">name</span> <span class="o">=</span> <span class="n">root</span><span class="o">.</span><span class="na">findViewById</span><span class="o">(</span><span class="no">R</span><span class="o">.</span><span class="na">id</span><span class="o">.</span><span class="na">name</span><span class="o">);</span>
<span class="k">if</span> <span class="o">(</span><span class="n">name</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">NullPointerException</span><span class="o">(</span><span class="s">"View 'name' required but not found"</span><span class="o">);</span>
<span class="o">}</span>
<span class="nc">TextView</span> <span class="n">email</span> <span class="o">=</span> <span class="n">root</span><span class="o">.</span><span class="na">findViewById</span><span class="o">(</span><span class="no">R</span><span class="o">.</span><span class="na">id</span><span class="o">.</span><span class="na">email</span><span class="o">);</span>
<span class="k">if</span> <span class="o">(</span><span class="n">email</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">NullPointerException</span><span class="o">(</span><span class="s">"View 'email' required but not found"</span><span class="o">);</span>
<span class="o">}</span>
<span class="k">return</span> <span class="k">new</span> <span class="nf">MainBinding</span><span class="o">(</span><span class="n">root</span><span class="o">,</span> <span class="n">name</span><span class="o">,</span> <span class="n">email</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>If you compile, dex, and dump the contents of the <code class="highlighter-rouge">.dex</code> file using <a href="https://github.com/JesusFreke/smali">Baksmali</a>, you can see these strings in the string data section of the output.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> |[20] string_data_item
00044f: 22 | utf16_size = 34
000450: 5669 6577 2027 6e61| data = "View \'name\' required but not found"
000458: 6d65 2720 7265 7175|
000460: 6972 6564 2062 7574|
000468: 206e 6f74 2066 6f75|
000470: 6e64 00 |
|[21] string_data_item
000473: 23 | utf16_size = 35
000474: 5669 6577 2027 656d| data = "View \'email\' required but not found"
00047c: 6169 6c27 2072 6571|
000484: 7569 7265 6420 6275|
00048c: 7420 6e6f 7420 666f|
000494: 756e 6400 |
</code></pre></div></div>
<p>In order to be encoded in the dex file format, these strings require 36 and 37 bytes, respectively (the extra two bytes for each encode their length and a null terminator).</p>
<p>With some napkin math we can estimate the cost of these strings in a real app. Each string requires 32 bytes plus the length of the view ID which we’ll say is usually around 12 characters. A medium-sized app has around 50 layouts each with around 10 views. So 50 * 10 * (32 + 12) yields a total cost of 22KB. This isn’t a huge amount of space, but considering we expect these strings to never be used unless there’s a programming error the overhead feels unfortunate.</p>
<p>Strings are de-duplicated in dex so if the common parts of the string were separated we would only pay their cost once. Additionally, the string data section is also used to hold the names of fields so strings which match the name of a field will be free. Using this information, we might naively try to split up the string into three pieces.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> if (name == null) {
<span class="gd">- throw new NullPointerException("View 'name' required but not found");
</span><span class="gi">+ throw new NullPointerException("View '" + "name" + "' required but not found");
</span> }
TextView email = root.findViewById(R.id.email);
if (email == null) {
<span class="gd">- throw new NullPointerException("View 'email' required but not found");
</span><span class="gi">+ throw new NullPointerException("View '" + "email" + "' required but not found");
</span> }
</code></pre></div></div>
<p>Unfortunately, <code class="highlighter-rouge">javac</code> sees the concatenation of constants as something it can optimize so it turns them back into single, unique strings. To outsmart it, we need to generate code which uses a <code class="highlighter-rouge">StringBuilder</code> or the little-known <code class="highlighter-rouge">String.concat</code> method.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> if (name == null) {
<span class="gd">- throw new NullPointerException("View 'name' required but not found");
</span><span class="gi">+ throw new NullPointerException("Missing required view with ID: ".concat("name"));
</span> }
TextView email = root.findViewById(R.id.email);
if (email == null) {
<span class="gd">- throw new NullPointerException("View 'email' required but not found");
</span><span class="gi">+ throw new NullPointerException("Missing required view with ID: ".concat("email"));
</span> }
</code></pre></div></div>
<p>Now the dex file only contains a single prefix string and we don’t pay for the ID strings because they were already being used for the <code class="highlighter-rouge">R.id.</code> fields.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> |[17] string_data_item
00046a: 1f | utf16_size = 31
00046b: 4d69 7373 696e 6720| data = "Missing required view with ID: "
000473: 7265 7175 6972 6564|
00047b: 2076 6965 7720 7769|
000483: 7468 2049 443a 2000|
</code></pre></div></div>
<p>22KB of string data reduced to 33 bytes! Now it is worth noting that we spend an extra 7 bytes loading the second string and invoking <code class="highlighter-rouge">String.concat</code>, but since the string was always more than 32 bytes it’s still a nice win. There’s still room to de-duplicate the actual concatenation and exception throwing code so that it’s only paid once per class instead of once per view, but I’ll leave that for another post.</p>
<hr />
<p>Seeing either of these optimizations in manually written code should raise an eyebrow. The individual savings of applying them are not worth their otherwise unidiomatic nature. With code generation, however, the economics are different. A single change to the generator can have optimizations like this apply to hundreds or thousands of locations producing a much larger effect.</p>
R8 Optimization: Class Constant Operations2019-02-27T00:00:00+00:00https://jakewharton.com/r8-optimization-class-constant-operations<blockquote>
<p>Note: This post is part of a series on D8 and R8, Android’s new dexer and optimizer, respectively. For an intro to D8 read <a href="/androids-java-8-support/">“Android’s Java 8 support”</a>. For an intro to R8 read <a href="/r8-optimization-staticization/">“R8 Optimization: Staticization”</a>.</p>
</blockquote>
<p>The <a href="/r8-optimization-string-constant-operations/">previous post in the series</a> showed R8 (and D8) invoking string methods at compile-time when the inputs were all constants. R8 is able to do this because the content of constant strings is available inside the bytecode. That post also claimed that strings are the only non-primitive type that can be manipulated like this at compile-time.</p>
<p>There is, however, another object type that can be manipulated at compile-time: classes. Classes are templates for the instances we interact with at runtime. Since bytecode fundamentally exists to hold these templates, some operations on classes can thus be performed at compile time.</p>
<h3 id="log-tags">Log Tags</h3>
<p>There’s an ongoing debate (if you can even call it that) on the best way to define a tag string in a class. Historically there have been two strategies: string literals and calling <code class="highlighter-rouge">getSimpleName()</code> on the class.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">String</span> <span class="no">TAG</span> <span class="o">=</span> <span class="s">"MyClass"</span><span class="o">;</span>
<span class="c1">// or</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">String</span> <span class="no">TAG</span> <span class="o">=</span> <span class="nc">MyClass</span><span class="o">.</span><span class="na">class</span><span class="o">.</span><span class="na">getSimpleName</span><span class="o">();</span>
</code></pre></div></div>
<p>Let’s compare the difference in bytecode by defining both and adding some log messages.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">MyClass</span> <span class="o">{</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">String</span> <span class="no">TAG_STRING</span> <span class="o">=</span> <span class="s">"MyClass"</span><span class="o">;</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">String</span> <span class="no">TAG_CLASS</span> <span class="o">=</span> <span class="nc">MyClass</span><span class="o">.</span><span class="na">class</span><span class="o">.</span><span class="na">getSimpleName</span><span class="o">();</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">...</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">Log</span><span class="o">.</span><span class="na">d</span><span class="o">(</span><span class="no">TAG_STRING</span><span class="o">,</span> <span class="s">"String tag"</span><span class="o">);</span>
<span class="nc">Log</span><span class="o">.</span><span class="na">d</span><span class="o">(</span><span class="no">TAG_CLASS</span><span class="o">,</span> <span class="s">"Class tag"</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Compiling, dexing, and dumping the Dalvik bytecode shows the effect of the choice.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[000194] MyClass.<clinit>:()V
0000: const-class v0, LMyClass;
0002: invoke-virtual {v0}, Ljava/lang/Class;.getSimpleName:()Ljava/lang/String;
0005: move-result-object v0
0006: sput-object v0, LMyClass;.TAG_CLASS:Ljava/lang/String;
0008: return-void
[000120] MyClass.main:([Ljava/lang/String;)V
0000: const-string v1, "MyClass"
0002: const-string v0, "String tag"
0004: invoke-static {v1, v0}, Landroid/util/Log;.d:(Ljava/lang/String;Ljava/lang/String;)I
0007: sget-object v1, LMyClass;.a:Ljava/lang/String;
0009: const-string v0, "Class tag"
000b: invoke-static {v1, v0}, Landroid/util/Log;.d:(Ljava/lang/String;Ljava/lang/String;)I
000e: return-void
</code></pre></div></div>
<p>In the <code class="highlighter-rouge">main</code> method, index <code class="highlighter-rouge">0000</code> loads the constant string of the tag. Index <code class="highlighter-rouge">0007</code>, on the other hand, has to look up the static field in order to get the tag value. In the <code class="highlighter-rouge"><clinit></code> method, the static field is initialized by loading the <code class="highlighter-rouge">MyClass</code> class and then invoking <code class="highlighter-rouge">getSimpleName</code> at runtime. This method is automatically invoked the first time the class is loaded.</p>
<p>The string literal is more efficient but using the class reference is resilient to things like refactoring. But if you’ve read any of these posts so far, you should know where this is going! Let’s try again with R8 and look at its output.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[000120] MyClass.main:([Ljava/lang/String;)V
0000: const-string v1, "MyClass"
0002: const-string v0, "String tag"
0004: invoke-static {v1, v0}, Landroid/util/Log;.d:(Ljava/lang/String;Ljava/lang/String;)I
0007: const-string v0, "Class tag"
0009: invoke-static {v1, v0}, Landroid/util/Log;.d:(Ljava/lang/String;Ljava/lang/String;)I
000c: return-void
</code></pre></div></div>
<p>The bytecode which came after index <code class="highlighter-rouge">0004</code> that loaded the second tag has disappeared and <code class="highlighter-rouge">v1</code>, the string literal tag, was re-used for the second call to <code class="highlighter-rouge">Log</code>.</p>
<p>Since the simple name of <code class="highlighter-rouge">MyClass</code> is known at compile-time, R8 has replaced <code class="highlighter-rouge">MyClass.class.getSimpleName()</code> with the string literal <code class="highlighter-rouge">"MyClass"</code>. Because the field value is now a constant, the <code class="highlighter-rouge"><clinit></code> method becomes empty and is removed. At the usage site, the <code class="highlighter-rouge">sget-object</code> bytecode was replaced with a <code class="highlighter-rouge">const-string</code> for the constant. Finally, the two <code class="highlighter-rouge">const-string</code> bytecodes which reference the same string were de-duplicated and the value is reused.</p>
<p>So while the verdict might not be in on which pattern to use for log tag fields, R8 makes sure that those choosing the class-based route don’t incur any additional runtime overhead. And because the <code class="highlighter-rouge">getSimpleName()</code> computation is trivial, D8 will actually perform it as well!<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup></p>
<h3 id="applicability">Applicability</h3>
<p>Being able to compute <code class="highlighter-rouge">getSimpleName()</code> (and <code class="highlighter-rouge">getName()</code> and <code class="highlighter-rouge">getCanonicalName()</code> too!) on a <code class="highlighter-rouge">MyClass.class</code> reference seems of limited use–potentially even <em>solely</em> for this log tag case. The optimization only works with a class literal reference–<code class="highlighter-rouge">getClass()</code> won’t work! It is once again in combination with other R8 features that this optimization starts to apply more.</p>
<p>Consider a class which abstracts logging and uses a static initializer that accepts which class will be sending log messages.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Logger</span> <span class="o">{</span>
<span class="kd">static</span> <span class="nc">Logger</span> <span class="nf">get</span><span class="o">(</span><span class="nc">Class</span><span class="o"><?></span> <span class="n">cls</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="nf">Logger</span><span class="o">(</span><span class="n">cls</span><span class="o">.</span><span class="na">getSimpleName</span><span class="o">());</span>
<span class="o">}</span>
<span class="kd">private</span> <span class="nf">Logger</span><span class="o">(</span><span class="nc">String</span> <span class="n">tag</span><span class="o">)</span> <span class="o">{</span> <span class="cm">/* … */</span> <span class="o">}</span>
<span class="o">}</span>
<span class="kd">class</span> <span class="nc">MyClass</span> <span class="o">{</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">Logger</span> <span class="n">logger</span> <span class="o">=</span> <span class="nc">Logger</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="nc">MyClass</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
<span class="o">}</span>
</code></pre></div></div>
<p>If <code class="highlighter-rouge">Logger.get</code> is inlined to all of its call sites, the call to <code class="highlighter-rouge">Class.getSimpleName</code> which previously had a dynamic input from the method parameter will change to a static input of a class reference (<code class="highlighter-rouge">MyClass.class</code> in this case). R8 can now replace the call with a string literal resulting in a field initializer that directly invokes the constructor (which will also have its <code class="highlighter-rouge">private</code> modifier removed).</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">MyClass</span> <span class="o">{</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">Logger</span> <span class="n">logger</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">Logger</span><span class="o">(</span><span class="s">"MyClass"</span><span class="o">);</span>
<span class="o">}</span>
</code></pre></div></div>
<p>This relies on the <code class="highlighter-rouge">get</code> method being small enough or being called in a way that the heuristics of R8 will perform the inlining.</p>
<p>The Kotlin language offers the ability to force a function to be inlined. It also allows marking a generic type parameter on an inline fuction as “reified” which ensures that the compiler knows which class it resolves to when compiling. With these features we can ensure our function is always inlined and that <code class="highlighter-rouge">getSimpleName</code> is always called on an explicit class reference.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Logger</span> <span class="k">private</span> <span class="k">constructor</span><span class="p">(</span><span class="kd">val</span> <span class="py">tag</span><span class="p">:</span> <span class="nc">String</span><span class="p">)</span> <span class="p">{</span>
<span class="p">}</span>
<span class="k">inline</span> <span class="k">fun</span> <span class="p"><</span><span class="k">reified</span> <span class="nc">T</span> <span class="p">:</span> <span class="nc">Any</span><span class="p">></span> <span class="nf">logger</span><span class="p">()</span> <span class="p">=</span> <span class="nc">Logger</span><span class="p">(</span><span class="nc">T</span><span class="o">::</span><span class="k">class</span><span class="p">.</span><span class="n">java</span><span class="p">.</span><span class="n">simpleName</span><span class="p">)</span>
<span class="kd">class</span> <span class="nc">MyClass</span> <span class="p">{</span>
<span class="k">companion</span> <span class="k">object</span> <span class="p">{</span>
<span class="k">private</span> <span class="kd">val</span> <span class="py">logger</span> <span class="p">=</span> <span class="n">logger</span><span class="p"><</span><span class="nc">MyClass</span><span class="p">>()</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The initializer for <code class="highlighter-rouge">logger</code> will always have the bytecode equivalent of <code class="highlighter-rouge">MyClass.class.getSimpleName()</code> which R8 can then always replace with a string literal.</p>
<p>For other Kotlin examples, type inference can often allow omitting the explicit type parameter.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">inline</span> <span class="k">fun</span> <span class="p"><</span><span class="k">reified</span> <span class="nc">T</span><span class="p">></span> <span class="nf">typeAndValue</span><span class="p">(</span><span class="n">value</span><span class="p">:</span> <span class="nc">T</span><span class="p">)</span> <span class="p">=</span> <span class="s">"${T::class.java.name}: $value"</span>
<span class="k">fun</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span>
<span class="nf">println</span><span class="p">(</span><span class="nf">typeAndValue</span><span class="p">(</span><span class="s">"hey"</span><span class="p">))</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This example outputs “java.lang.String: hey” and its bytecode contains only two constant strings, a <code class="highlighter-rouge">StringBuilder</code> to concatenate them, and a call to <code class="highlighter-rouge">System.out.println</code>. And if <a href="https://issuetracker.google.com/issues/114002137">this issue</a> was implemented, you’d wind up with only a single string and the call to <code class="highlighter-rouge">System.out.println</code>.</p>
<h3 id="obfuscation-and-optimization">Obfuscation and Optimization</h3>
<p>Since this optimization operates on classes, it has to interact with the other features of R8 that might affect a class such as obfuscation and different optimizations.</p>
<p>Let’s go back to the original example.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">MyClass</span> <span class="o">{</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">String</span> <span class="no">TAG_STRING</span> <span class="o">=</span> <span class="s">"MyClass"</span><span class="o">;</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">String</span> <span class="no">TAG_CLASS</span> <span class="o">=</span> <span class="nc">MyClass</span><span class="o">.</span><span class="na">class</span><span class="o">.</span><span class="na">getSimpleName</span><span class="o">();</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">...</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">Log</span><span class="o">.</span><span class="na">d</span><span class="o">(</span><span class="no">TAG_STRING</span><span class="o">,</span> <span class="s">"String tag"</span><span class="o">);</span>
<span class="nc">Log</span><span class="o">.</span><span class="na">d</span><span class="o">(</span><span class="no">TAG_CLASS</span><span class="o">,</span> <span class="s">"Class tag"</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>What happens if this class is obfuscated? If R8 was not replacing the <code class="highlighter-rouge">getSimpleName</code> call, the first log message would have a tag of “MyClass” and the second would have a tag matching the obfuscated class name such as “a”.</p>
<p>In order for R8 to be allowed to replace <code class="highlighter-rouge">getSimpleName</code> it needs to do so with a value that matches what the behavior would be at runtime. Thankfully, since R8 is also the tool which is performing obfuscation, it can defer the replacement until the the class has been given its final name.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[000158] a.main:([Ljava/lang/String;)V
0000: const-string v1, "MyClass"
0002: const-string v0, "String tag"
0004: invoke-static {v1, v0}, Landroid/util/Log;.d:(Ljava/lang/String;Ljava/lang/String;)I
0007: const-string v1, "a"
0009: const-string v0, "Class tag"
000b: invoke-static {v1, v0}, Landroid/util/Log;.d:(Ljava/lang/String;Ljava/lang/String;)I
000e: return-void
</code></pre></div></div>
<p>Note how index <code class="highlighter-rouge">0007</code> now will load a tag value for the second log call (unlike the original R8 output) and how it correctly reflects the obfuscated name.</p>
<p>There are other R8 optimizations which affect the class name even when obfuscation is disabled. While I plan to cover it in a future post, R8 will sometimes merge a superclass into a subtype if it can prove the superclass isn’t needed and the subtype is the only one. When this happens, the class name string optimization will correctly reflect the subtype name even if the original code was equivalent to <code class="highlighter-rouge">TheSupertype.class.getSimpleName()</code>.</p>
<h3 id="string-data-section">String Data Section</h3>
<p><a href="/r8-optimization-string-constant-operations/#money-left-on-the-table">The previous post</a> talked about how performing an operation like <code class="highlighter-rouge">String.substring</code> or string concatenation at compile-time could lead to the string section of the dex file increasing in size.<sup id="fnref:2"><a href="#fn:2" class="footnote">2</a></sup> The optimization in this post produces strings which might not otherwise exist so that is also a possibility here.</p>
<p>There’s two cases to consider: when obfuscation is enabled and when it is disabled.</p>
<p>When obfuscation is enabled calls to <code class="highlighter-rouge">getSimpleName()</code> should not create a new string. Both classes and methods will be obfuscated using the same dictionary which by default starts with single letters. This means that for an obfuscated class named <code class="highlighter-rouge">b</code>, inserting the string “b” is almost always free since there is going to be a method or field whose name is also <code class="highlighter-rouge">b</code>. In the dex file all strings are stored in a single pool which contains the literals, class names, method names, and field names making the probability of a match when obfuscating very high.</p>
<p>With obfuscation disabled, though, replacing <code class="highlighter-rouge">getSimpleName()</code> is never free. Despite the unified string section of the dex file, class names are stored in <a href="https://source.android.com/devices/tech/dalvik/dex-format#typedescriptor">type descriptor form</a>. This includes the package name, uses <code class="highlighter-rouge">/</code> as separators, and is prefixed with <code class="highlighter-rouge">L</code> and suffixed with <code class="highlighter-rouge">;</code>. For <code class="highlighter-rouge">MyClass</code>, if in a hypothetical <code class="highlighter-rouge">com.example</code> package, the string data contains an entry for <code class="highlighter-rouge">Lcom/example/MyClass;</code>. Because of this format, the string “MyClass” doesn’t already exist and will need to be added.</p>
<p>Both <code class="highlighter-rouge">getName()</code> and <code class="highlighter-rouge">getCanonicalName()</code> will also, unfortunately, always create new strings. Even though these return a fully-qualified strings, they don’t match the type descriptor form which is already present in string data.</p>
<p>Since this optimization has the potential to create a large amount of strings, it’s currently disabled for everything except top-level types. This means that it works in the <code class="highlighter-rouge">MyClass</code> example from this post but in a nested type or anonymous type it will not apply. There is also some escape analysis done to avoid applying the optimization for calls inside a single method. Both of these minimize any adverse impact on your dex size.</p>
<hr />
<p>The next post on R8 will look at an optimization which produces class literals like those used in this post (i.e., the <code class="highlighter-rouge">const-class</code> bytecodes created from <code class="highlighter-rouge">MyClass.class</code>). You won’t be surprised when that post shows class literal creation which in turn allows the optimizations from this post to apply which in turn allows the string optimizations to apply and so on.</p>
<p><em>(This post was adapted from a part of my <a href="/digging-into-d8-and-r8">Digging into D8 and R8</a> talk that was never presented. Watch the video and look out for future blog posts for more content like this.)</em></p>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>It won’t, however, replace the <code class="highlighter-rouge">sget-object</code> bytecodes with <code class="highlighter-rouge">const-string</code> nor remove the now-empty <code class="highlighter-rouge"><clinit></code> method. <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
<li id="fn:2">
<p>Coincidentally, compile-time <code class="highlighter-rouge">substring()</code> computation <a href="https://r8-review.googlesource.com/c/r8/+/34260/">landed in R8</a> yesterday! <a href="#fnref:2" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
R8 Optimization: String Constant Operations2019-02-12T00:00:00+00:00https://jakewharton.com/r8-optimization-string-constant-operations<blockquote>
<p>Note: This post is part of a series on D8 and R8, Android’s new dexer and optimizer, respectively. For an intro to D8 read <a href="/androids-java-8-support/">“Android’s Java 8 support”</a>. For an intro to R8 read <a href="/r8-optimization-staticization/">“R8 Optimization: Staticization”</a>.</p>
</blockquote>
<p>The <a href="/r8-optimization-value-assumption/">previous post in the series</a> covered an R8 flag which allows you to specify the return value range of a field or method. R8 can use this to automatically remove conditionals against <code class="highlighter-rouge">SDK_INT</code> based on your app’s minimum supported API level, for example. That can only happen because multiple R8 features are working together. This post (and the next few) will cover smaller optimizations of R8 which work best when combined with others.</p>
<p>Aside from the eight primitive types of Java, all of the other values your program interacts with are instances of classes whose data can only be manipulated at runtime. That is, all except for one type: strings. Strings are such a fundamental and ubiquitous type that they are given special treatment in the Java and Kotlin language, in Java bytecode, and in Dalvik bytecode. And because of that special treatment, tools like R8 can manipulate them at compile-time!</p>
<h3 id="constant-pool-and-string-data">Constant Pool and String Data</h3>
<p>When you write a string literal in Java or Kotlin, the contents of that string are encoded in a special section of the bytecode. For Java bytecode it’s called the <em>constant pool</em>. For Dalvik bytecode it’s called the <em>string data section</em>. In addition to string literals which were present in the source code, strings for the names of types, methods, fields, and other structural elements are included in these sections.</p>
<p>When you look at the Java bytecode of a class file through <code class="highlighter-rouge">javap</code> as these posts have been doing, references to the constant pool use an octothorpe (<code class="highlighter-rouge">#</code>) followed by a number.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>0: new #2 // class java/lang/StringBuilder
3: dup
4: invokespecial #3 // Method java/lang/StringBuilder."<init>":()V
7: ldc #4 // String A:
</code></pre></div></div>
<p>Helpful comments are included so that we don’t have to manually consult the constant pool to figure out what each means.</p>
<p>If you invoke <code class="highlighter-rouge">javap</code> with the <code class="highlighter-rouge">-v</code> argument the constant pool will be included in the output.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Constant pool:
#1 = Methodref #9.#18 // java/lang/Object."<init>":()V
#2 = Class #19 // java/lang/StringBuilder
#3 = Methodref #2.#18 // java/lang/StringBuilder."<init>":()V
#4 = String #20 // A:
⋮
#10 = Utf8 <init>
#11 = Utf8 ()V
⋮
#18 = NameAndType #10:#11 // "<init>":()V
#19 = Utf8 java/lang/StringBuilder
#20 = Utf8 A:
</code></pre></div></div>
<p>#4 is a <code class="highlighter-rouge">String</code> type whose data is at #20 which is a UTF-8 entry for “A:”. This was one of the string literals from the source (taken from <a href="/androids-java-9-10-11-and-12-support/#string-concat">the Java 9 string concat example</a>). If you look at #2 or #3, they’re signatures for a <code class="highlighter-rouge">Class</code> and <code class="highlighter-rouge">Methodref</code> (method reference), respectively. Each uses one or more UTF-8 entries to create the signature it represents.</p>
<p>When using <code class="highlighter-rouge">dexdump</code> to look at Dalvik bytecode, the program doesn’t show the string data section directly. Instead, strings are substituted into the bytecode output to make it easier to read.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>0000: new-instance v0, Ljava/lang/StringBuilder; // type@0003
0002: invoke-direct {v0}, Ljava/lang/StringBuilder;.<init>:()V // method@0003
0005: const-string v1, "A: " // string@0002
</code></pre></div></div>
<p>Hints of the string data section are shown in the comments which follow each line. <code class="highlighter-rouge">string@0002</code> indicates this literal comes from index 2 in the string data section. The <code class="highlighter-rouge">type@0003</code> and <code class="highlighter-rouge">method@0003</code> hints point to separate sections of the dex which themselves eventually use the string data to create their signatures (similar to how the constant pool in the Java bytecode worked).</p>
<h3 id="string-operations">String Operations</h3>
<p>Performing string operations on literals isn’t something that frequently happens in your source code. You wouldn’t write something like <code class="highlighter-rouge">new User("OliveJakeHazel".substring(5, 9))</code> to create a <code class="highlighter-rouge">User</code> named “Jake”. You would use <code class="highlighter-rouge">"Jake"</code> as the string literal without a <code class="highlighter-rouge">substring</code> call. One notable exception to this is computing the length of a string literal.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">static</span> <span class="nc">String</span> <span class="nf">patternHost</span><span class="o">(</span><span class="nc">String</span> <span class="n">pattern</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">pattern</span><span class="o">.</span><span class="na">startsWith</span><span class="o">(</span><span class="no">WILDCARD</span><span class="o">)</span>
<span class="o">?</span> <span class="n">pattern</span><span class="o">.</span><span class="na">substring</span><span class="o">(</span><span class="no">WILDCARD</span><span class="o">.</span><span class="na">length</span><span class="o">())</span>
<span class="o">:</span> <span class="n">pattern</span><span class="o">;</span>
<span class="o">}</span>
</code></pre></div></div>
<p>This code is adapted from a real example inside <a href="https://github.com/square/okhttp">OkHttp</a> where a string is tested for a prefix and then conditionally removed. The length is computed so that if the constant changes the value passed to <code class="highlighter-rouge">substring</code> remains correct.</p>
<p>Let’s take a look at what Dalvik bytecode this example produces.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[0001a8] Test.patternHost:(Ljava/lang/String;)Ljava/lang/String;
0000: const-string v0, "*."
0002: invoke-virtual {v2, v0}, Ljava/lang/String;.startsWith:(Ljava/lang/String;)Z
0005: move-result v1
0006: if-eqz v1, 0010
0008: invoke-virtual {v0}, Ljava/lang/String;.length:()I
0011: move-result v1
0012: invoke-virtual {v2, v1}, Ljava/lang/String;.substring:(I)Ljava/lang/String;
000f: move-result-object v2
0010: return-object v2
</code></pre></div></div>
<p>In index <code class="highlighter-rouge">0000</code> to <code class="highlighter-rouge">0002</code>, the <code class="highlighter-rouge">WILDCARD</code> constant (whose value is the literal <code class="highlighter-rouge">"*."</code>) is loaded into register <code class="highlighter-rouge">v0</code> in order to call <code class="highlighter-rouge">startWith</code> on the parameter (in <code class="highlighter-rouge">v2</code>). Later, in index <code class="highlighter-rouge">0008</code> to <code class="highlighter-rouge">0011</code>, the length of <code class="highlighter-rouge">v0</code> is calculated and stored in <code class="highlighter-rouge">v1</code> so that it can be used to call <code class="highlighter-rouge">substring</code> on the parameter.</p>
<p>Since <code class="highlighter-rouge">WILDCARD</code> is a constant initialized with a string literal, its length is also a constant. Computing its length at runtime is a waste of time because it will always produce the same value. When the above code is compiled with R8, the call to <code class="highlighter-rouge">length()</code> on a constant is replaced with the value as determined at compile-time.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[0001a8] Test.patternHost:(Ljava/lang/String;)Ljava/lang/String;
0000: const-string v0, "*."
0002: invoke-virtual {v1, v0}, Ljava/lang/String;.startsWith:(Ljava/lang/String;)Z
0005: move-result v0
0006: if-eqz v0, 000d
0008: const/4 v0, #int 2
0009: invoke-virtual {v1, v0}, Ljava/lang/String;.substring:(I)Ljava/lang/String;
000c: move-result-object v1
000d: return-object v1
</code></pre></div></div>
<p>Index <code class="highlighter-rouge">0008</code> now loads the constant value of 2 which is immediately passed to the <code class="highlighter-rouge">substring</code> call. The bytecode gets the performance benefit of a hardcoded value without the maintenance burden of keeping the two values in sync in the source code.</p>
<p>And because this computation was trivial and removing the call to <code class="highlighter-rouge">length()</code> won’t change the program’s behavior, D8 will also perform this optimization!</p>
<h3 id="inlining">Inlining</h3>
<p>Computing the length of a constant string isn’t the only string operation that can happen at compile-time. Common string operations such as <code class="highlighter-rouge">startWith</code>, <code class="highlighter-rouge">indexOf</code>, and <code class="highlighter-rouge">substring</code> can all be computed provided that their arguments are also constants. While this is rare to find verbatim in source code, method inlining can create situations where this happens.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Test</span> <span class="o">{</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">String</span> <span class="no">WILDCARD</span> <span class="o">=</span> <span class="s">"*."</span><span class="o">;</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="nc">String</span> <span class="nf">patternHost</span><span class="o">(</span><span class="nc">String</span> <span class="n">pattern</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">pattern</span><span class="o">.</span><span class="na">startsWith</span><span class="o">(</span><span class="no">WILDCARD</span><span class="o">)</span>
<span class="o">?</span> <span class="n">pattern</span><span class="o">.</span><span class="na">substring</span><span class="o">(</span><span class="no">WILDCARD</span><span class="o">.</span><span class="na">length</span><span class="o">())</span>
<span class="o">:</span> <span class="n">pattern</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="nc">String</span> <span class="nf">canonicalHost</span><span class="o">(</span><span class="nc">String</span> <span class="n">pattern</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">String</span> <span class="n">host</span> <span class="o">=</span> <span class="n">patternHost</span><span class="o">(</span><span class="n">pattern</span><span class="o">);</span>
<span class="k">return</span> <span class="nc">HttpUrl</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="s">"http://"</span> <span class="o">+</span> <span class="n">host</span><span class="o">).</span><span class="na">host</span><span class="o">();</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">...</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">String</span> <span class="n">pattern</span> <span class="o">=</span> <span class="s">"*.example.com"</span><span class="o">;</span>
<span class="nc">String</span> <span class="n">canonical</span> <span class="o">=</span> <span class="n">canonicalHost</span><span class="o">(</span><span class="n">pattern</span><span class="o">);</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">canonical</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Take this more complete example where the <code class="highlighter-rouge">main</code> method calls a public library method <code class="highlighter-rouge">canonicalHost</code> with a string literal. The <code class="highlighter-rouge">canonicalHost</code> library method delegates to <code class="highlighter-rouge">patternHost</code> which is a private library method. Because this program is so small both methods will ultimately be inlined into the <code class="highlighter-rouge">main</code> method.</p>
<p>We can pretend this inlining happened at the source-level to see how the code changes as the string optimizations apply.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Test</span> <span class="o">{</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">String</span> <span class="no">WILDCARD</span> <span class="o">=</span> <span class="s">"*."</span><span class="o">;</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">...</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">String</span> <span class="n">pattern</span> <span class="o">=</span> <span class="s">"*.example.com"</span><span class="o">;</span>
<span class="nc">String</span> <span class="n">host</span> <span class="o">=</span> <span class="n">pattern</span><span class="o">.</span><span class="na">startsWith</span><span class="o">(</span><span class="no">WILDCARD</span><span class="o">)</span>
<span class="o">?</span> <span class="n">pattern</span><span class="o">.</span><span class="na">substring</span><span class="o">(</span><span class="no">WILDCARD</span><span class="o">.</span><span class="na">length</span><span class="o">())</span>
<span class="o">:</span> <span class="n">pattern</span><span class="o">;</span>
<span class="nc">String</span> <span class="n">canonical</span> <span class="o">=</span> <span class="nc">HttpUrl</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="s">"http://"</span> <span class="o">+</span> <span class="n">host</span><span class="o">).</span><span class="na">host</span><span class="o">();</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">canonical</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>R8’s intermediate representation (IR) during compilation uses static single-assignment form (SSA) (<a href="/r8-optimization-null-data-flow-analysis-part-1/">introduced in part 1 of the null analysis</a>) which allows it to, among other things, trace the origin of local variables. Despite <code class="highlighter-rouge">startsWith</code> operating on the variable <code class="highlighter-rouge">pattern</code>, that variable’s origin can be traced to the string literal <code class="highlighter-rouge">"*.example.com"</code>. The argument to <code class="highlighter-rouge">startsWith</code>, <code class="highlighter-rouge">WILDCARD</code>, is also a string constant allowing the whole operation to be replaced with its result at compile-time.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> String pattern = "*.example.com";
<span class="gd">-String host = pattern.startsWith(WILDCARD)
</span><span class="gi">+String host = true
</span> ? pattern.substring(WILDCARD.length())
</code></pre></div></div>
<p>Dead-code elimination removes the impossible ‘else’ branch and the conditional.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> String pattern = "*.example.com";
<span class="gd">-String host = true
- ? pattern.substring(WILDCARD.length())
- : pattern;
</span><span class="gi">+String host = pattern.substring(WILDCARD.length());
</span> String canonical = HttpUrl.get("http://" + host).host();
</code></pre></div></div>
<p>The call to <code class="highlighter-rouge">length()</code> on a string constant is replaced with the constant integer value as demonstrated in the previous section.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> String pattern = "*.example.com";
<span class="gd">-String host = pattern.substring(WILDCARD.length());
</span><span class="gi">+String host = pattern.substring(2);
</span> String canonical = HttpUrl.get("http://" + host).host();
</code></pre></div></div>
<p>Compiling and dexing the original three-method example with R8 confirms that this is the final result.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ javac -cp okhttp-3.13.1.jar Test.java
$ cat rules.txt
-keepclasseswithmembers class * {
public static void main(java.lang.String[]);
}
$ java -jar r8.jar \
--lib $ANDROID_HOME/platforms/android-28/android.jar \
--release \
--output . \
--pg-conf rules.txt \
*.class
$ $ANDROID_HOME/build-tools/28.0.3/dexdump -d classes.dex
[0001c0] Test.main:([Ljava/lang/String;)V
0000: const/4 v2, #int 2
0001: const-string v0, "*.example.com"
0003: invoke-virtual {v0, v2}, Ljava/lang/String;.substring:(I)Ljava/lang/String;
0006: move-result-object v2
0007: new-instance v0, Ljava/lang/StringBuilder;
0009: invoke-direct {v0}, Ljava/lang/StringBuilder;.<init>:()V
000c: const-string v1, "http://"
000e: invoke-virtual {v0, v1}, Ljava/lang/StringBuilder;.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
0011: invoke-virtual {v0, v2}, Ljava/lang/StringBuilder;.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
0014: invoke-virtual {v0}, Ljava/lang/StringBuilder;.toString:()Ljava/lang/String;
0017: move-result-object v2
0018: invoke-static {v2}, Lokhttp3/HttpUrl;.get:(Ljava/lang/String;)Lokhttp3/HttpUrl;
001b: move-result-object v2
001c: invoke-virtual {v2}, Lokhttp3/HttpUrl;.host:()Ljava/lang/String;
001f: move-result-object v2
0020: sget-object v0, Ljava/lang/System;.out:Ljava/io/PrintStream;
0022: invoke-virtual {v0, v2}, Ljava/io/PrintStream;.println:(Ljava/lang/String;)V
0025: return-void
</code></pre></div></div>
<p>The <code class="highlighter-rouge">startsWith</code> check and conditional have been removed because inlining has made the receiver string available as a constant. Our dex file is a bit smaller and our program runs a bit faster now because this condition which always produced the same value was computed at compile-time.</p>
<h3 id="money-left-on-the-table">Money Left on the Table</h3>
<p>Having <code class="highlighter-rouge">length()</code> and <code class="highlighter-rouge">startsWith()</code> replaced with a value computed at compile-time is a nice win. Other methods on <code class="highlighter-rouge">String</code> can be computed at compile-time such as <code class="highlighter-rouge">isEmpty()</code>, <code class="highlighter-rouge">contains()</code>, <code class="highlighter-rouge">endsWith()</code>, <code class="highlighter-rouge">equals()</code>, and <code class="highlighter-rouge">equalsIgnoreCase()</code>. Looking at the result above leaves me unsatisfied because optimizations were left on the table. Let’s look at the final form as if it were source code and analyze what <em>didn’t</em> happen.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">String</span> <span class="n">pattern</span> <span class="o">=</span> <span class="s">"*.example.com"</span><span class="o">;</span>
<span class="nc">String</span> <span class="n">host</span> <span class="o">=</span> <span class="n">pattern</span><span class="o">.</span><span class="na">substring</span><span class="o">(</span><span class="mi">2</span><span class="o">);</span>
<span class="nc">String</span> <span class="n">canonical</span> <span class="o">=</span> <span class="nc">HttpUrl</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="s">"http://"</span> <span class="o">+</span> <span class="n">host</span><span class="o">).</span><span class="na">host</span><span class="o">();</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">canonical</span><span class="o">);</span>
</code></pre></div></div>
<p>The now-removed call to <code class="highlighter-rouge">startsWith</code> was able to be eliminated because the receiver (i.e., the target string) and argument were both known at compile-time. Looking at the above example, that condition holds true for the call to <code class="highlighter-rouge">substring</code>. It should have been eliminated.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gd">-String pattern = "*.example.com";
-String host = pattern.substring(2);
</span><span class="gi">+String host = "example.com";
</span> String canonical = HttpUrl.get("http://" + host).host();
</code></pre></div></div>
<p>The argument sent to <code class="highlighter-rouge">HttpUrl.get</code> is now the result of string concatenation of two string literals. The need to concatenate those at runtime should have been eliminated.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gd">-String host = "example.com";
-String canonical = HttpUrl.get("http://" + host).host();
</span><span class="gi">+String canonical = HttpUrl.get("http://example.com").host();
</span></code></pre></div></div>
<p>These optimizations are likely to be included in a future version of R8 but they’re not as trivial as they might seem.</p>
<p>Every existing string optimization returns a primitive value such as a <code class="highlighter-rouge">boolean</code> or <code class="highlighter-rouge">int</code> which can be represented directly in the bytecode. As a result of those optimizations, it’s possible for the string data section to shrink if a string becomes unused. In the example above, <code class="highlighter-rouge">WILDCARD</code> becomes unused since its only two uses (as an argument to <code class="highlighter-rouge">startsWith</code> and as a receiver for <code class="highlighter-rouge">length</code>) were replaced with primitives and so it does not appear in the final dex file.</p>
<p>Computing a substring or performing concatenation at compile-time has the potential to increase the size of the string data section. If the input strings are still used in other parts of the application they won’t be eliminated. The new string, however, will always be added.</p>
<p>Doing these optimizations on the trivial program in this post removes 16 bytes of bytecode but adds 18 bytes of string data. In this case, because the input strings are not used anywhere else, an additional 20 bytes is removed for a net reduction of 18 bytes (ignoring the other parts of a dex).</p>
<p>In real-world applications it becomes less clear whether computing these is the correct choice. For now, these optimizations are not performed.</p>
<hr />
<p>When combined with inlining, R8’s string optimizations help eliminate dead code and improve runtime performance when working with string literals. To track updates to and show support for new <code class="highlighter-rouge">String</code> methods being computed at compile-time star <a href="https://issuetracker.google.com/issues/119364907">issuetracker.google.com/issues/119364907</a>. For string concatenation star <a href="https://issuetracker.google.com/issues/114002137">issuetracker.google.com/issues/114002137</a>.</p>
<p>The next post in the series will look at an optimization that creates string literals at compile-time which otherwise would need to be created at runtime.</p>
<p><em>(This post was adapted from a part of my <a href="/digging-into-d8-and-r8">Digging into D8 and R8</a> talk that was never presented. Watch the video and look out for future blog posts for more content like this.)</em></p>
R8 Optimization: Value Assumption2019-01-22T00:00:00+00:00https://jakewharton.com/r8-optimization-value-assumption<blockquote>
<p>Note: This post is part of a series on D8 and R8, Android’s new dexer and optimizer, respectively. For an intro to D8 read <a href="/androids-java-8-support/">“Android’s Java 8 support”</a>. For an intro to R8 read <a href="/r8-optimization-staticization/">“R8 Optimization: Staticization”</a>.</p>
</blockquote>
<p>The previous post (<a href="/r8-optimization-null-data-flow-analysis-part-1/">part 1</a>, <a href="/r8-optimization-null-data-flow-analysis-part-2/">part 2</a>) featured R8 performing data-flow analysis of variables in order to determine if they were maybe null, always null, or never null, and then potentially performing dead-code elimination based on that info.</p>
<p>Another way to think about that optimization is that R8 tracks the use of a variable along with a range of its possible nullability. If any conditional against that range can be determined to always produce the same result, dead-code elimination removes the unused branches and the conditional disappears. <a href="/r8-optimization-null-data-flow-analysis-part-2/#no-inlining-required">Part 2 of the last post</a> ended with an example where an <code class="highlighter-rouge">args</code> variable was passed into a <code class="highlighter-rouge">first</code> method and then checked for null before printing.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">first</span><span class="o">(</span><span class="n">args</span><span class="o">));</span>
<span class="k">if</span> <span class="o">(</span><span class="n">args</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"null!"</span><span class="o">);</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The range of nullability for <code class="highlighter-rouge">args</code> in that snippet is <code class="highlighter-rouge">[null, non-null]</code> (meaning its either null or a non-null reference).</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">first</span><span class="o">(</span><span class="n">args</span><span class="cm">/* [null, non-null] */</span><span class="o">));</span>
<span class="k">if</span> <span class="o">(</span><span class="n">args</span><span class="cm">/* [null, non-null] */</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"null!"</span><span class="o">);</span>
<span class="o">}</span>
</code></pre></div></div>
<p>In this state, R8 can’t do anything to the conditional because the reference might actually be null. However, if the <code class="highlighter-rouge">first</code> method checks its argument for null and throws an exception (as it did in that post), null can be eliminated as a possible value <em>after</em> the method call.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">first</span><span class="o">(</span><span class="n">args</span><span class="cm">/* [null, non-null] */</span><span class="o">));</span>
<span class="k">if</span> <span class="o">(</span><span class="n">args</span><span class="cm">/* [non-null] */</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"null!"</span><span class="o">);</span>
<span class="o">}</span>
</code></pre></div></div>
<p>With <code class="highlighter-rouge">args</code> only able to be a non-null reference at the time of the <code class="highlighter-rouge">if</code> check against null, the conditional will always be false and can be removed by normal dead-code elimination.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">first</span><span class="o">(</span><span class="n">args</span><span class="cm">/* [null, non-null] */</span><span class="o">));</span>
<span class="k">if</span> <span class="o">(</span><span class="kc">false</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"null!"</span><span class="o">);</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Right now this range tracking doesn’t extend beyond nullability. Checking an integer for being positive twice in a method does not cause R8 to eliminate the second conditional. That being said, there is a way to manually help R8 understand the range of other types.</p>
<h3 id="value-assumption">Value Assumption</h3>
<p>R8 uses the same configuration syntax as ProGuard in order to simplify migration. Once you’ve migrated, though, there are some R8-specific flags you can specify. This post deals with one of those flags: <code class="highlighter-rouge">-assumevalues</code>.<sup id="fnref:proguard-assumevalues"><a href="#fn:proguard-assumevalues" class="footnote">1</a></sup></p>
<p>The <code class="highlighter-rouge">-assumevalues</code> flag informs R8 that the specified field value or method’s return value will always be between a certain range or equal to a single value. The paragraph above mentioned that R8 won’t eliminate a second check for a positive value like it would a second check for null. If the integer value being checked comes from a method or is stored in a field this flag can help.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Count</span> <span class="o">{</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">...</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
<span class="n">count</span> <span class="o">=</span> <span class="mi">3</span><span class="o">;</span>
<span class="n">sayHi</span><span class="o">();</span>
<span class="o">}</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kt">int</span> <span class="n">count</span> <span class="o">=</span> <span class="mi">1</span><span class="o">;</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">sayHi</span><span class="o">()</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span><span class="n">count</span> <span class="o"><</span> <span class="mi">0</span><span class="o">)</span> <span class="o">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">IllegalStateException</span><span class="o">();</span>
<span class="o">}</span>
<span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">count</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"Hi!"</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>This example has a static field that dictates how many times “Hi!” is printed. Compiling, dexing with R8, and dumping the resulting bytecode shows that the check for negative remains in the bytecode despite being an impossible condition.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ javac *.java
$ cat rules.txt
-keepclasseswithmembers class * {
public static void main(java.lang.String[]);
}
-dontobfuscate
$ java -jar r8.jar \
--lib $ANDROID_HOME/platforms/android-28/android.jar \
--release \
--output . \
--pg-conf rules.txt \
*.class
$ $ANDROID_HOME/build-tools/28.0.3/dexdump -d classes.dex
[000148] Count.main:([Ljava/lang/String;)V
0000: const/4 v2, #int 3
0001: sput v2, LCount;.count:I
0003: sget v2, LCount;.count:I
0005: if-ltz v2, 0017
0007: const/4 v2, #int 0
0008: sget v0, LCount;.count:I
000a: if-ge v2, v0, 0016
000c: sget-object v0, Ljava/lang/System;.out:Ljava/io/PrintStream;
000e: const-string v1, "Hi!"
0010: invoke-virtual {v0, v1}, Ljava/io/PrintStream;.println:(Ljava/lang/String;)V
0013: add-int/lit8 v2, v2, #int 1
0015: goto 0008
0016: return-void
0017: new-instance v2, Ljava/lang/IllegalStateException;
0019: invoke-direct {v2}, Ljava/lang/IllegalStateException;.<init>:()V
001c: throw v2
</code></pre></div></div>
<p>R8 has inlined <code class="highlighter-rouge">sayHi()</code> into <code class="highlighter-rouge">main()</code> but everything is still here. Bytecode index 0000-0001 assign the value of 3 to <code class="highlighter-rouge">count</code>. Then index 0003-0005 read <code class="highlighter-rouge">count</code> and check if it’s less than 0, jumping to index 0017 if so. Index 0007-00015 is the loop, 0016 is the implicit <code class="highlighter-rouge">return</code>, and 0017 is the exception code (notice how it’s been moved to the bottom as explained in the previous post).</p>
<p>In order for R8 to eliminate the negative check it would need to analyze how the entire program interacts with <code class="highlighter-rouge">count</code>. While it would be trivial in this tiny example, in a real program the complexity of this task makes it infeasible.</p>
<p>Since this is application code in our control, we have additional knowledge of the domain of <code class="highlighter-rouge">count</code> which R8 can’t infer. Adding an <code class="highlighter-rouge">-assumevalues</code> flag to our <code class="highlighter-rouge">rules.txt</code> gives R8 the expected range of values that reading <code class="highlighter-rouge">count</code> will produce.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> -keepclasseswithmembers class * {
public static void main(java.lang.String[]);
}
-dontobfuscate
<span class="gi">+-assumevalues class Count {
+ static int count return 0..2147483647;
+}
</span></code></pre></div></div>
<p>Just as it did for tracking whether or not a reference could be null, R8 can now track the range of values of <code class="highlighter-rouge">count</code>.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="o">(</span><span class="n">count</span><span class="cm">/* [0..2147483647] */</span> <span class="o"><</span> <span class="mi">0</span><span class="o">)</span> <span class="o">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">IllegalStateException</span><span class="o">();</span>
<span class="o">}</span>
<span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">count</span><span class="cm">/* [0..2147483647] */</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"Hi!"</span><span class="o">);</span>
<span class="o">}</span>
</code></pre></div></div>
<p>With <code class="highlighter-rouge">count</code> only able to be a positive value at the time of the <code class="highlighter-rouge">if</code> check for negative, the conditional will always be false and can be removed by normal dead-code elimination.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="o">(</span><span class="kc">false</span><span class="o">)</span> <span class="o">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">IllegalStateException</span><span class="o">();</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Running R8 with the new <code class="highlighter-rouge">rules.txt</code> validates that this works.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[000128] Count.main:([Ljava/lang/String;)V
0000: const/4 v2, #int 3
0001: sput v2, LCount;.count:I
0003: sget v2, LCount;.count:I
0005: const/4 v2, #int 0
0006: sget v0, LCount;.count:I
0008: if-ge v2, v0, 0014
000a: sget-object v0, Ljava/lang/System;.out:Ljava/io/PrintStream;
000c: const-string v1, "Hi!"
000e: invoke-virtual {v0, v1}, Ljava/io/PrintStream;.println:(Ljava/lang/String;)V
0011: add-int/lit8 v2, v2, #int 1
0013: goto 0006
0014: return-void
</code></pre></div></div>
<p>Bytecode index 0000-0001 is still the assignment, 0005-0013 is the loop, and 0014 is the implicit <code class="highlighter-rouge">return</code>. No conditional in sight!</p>
<h4 id="side-effects">Side-Effects</h4>
<p>In the final bytecode from the previous example, index 0003 still reads <code class="highlighter-rouge">count</code> despite its value never actually being used (it’s immediately overwritten with 0 by the very next bytecode). This is the field read that would have been used for the now-eliminated conditional. Previous posts showed R8 eliminating unused code like this using its static, single-assignment intermediate representation (SSA IR). Why isn’t that happening here?</p>
<p>When R8 eliminates code based on <code class="highlighter-rouge">-assumevalues</code> it explicitly keeps the method call or field read despite not needing the value. A method call might trigger some other side-effect which would result in a behavior change if removed. A field read might cause a class to be loaded for the first time where a static initializer could have side-effects. It’s usually unlikely that your application has these side-effects or that you rely on them. Changing the rule from <code class="highlighter-rouge">-assumevalues</code> to <code class="highlighter-rouge">-assumenosideeffects</code> assures R8 of this allowing index 0003 to be removed.<sup id="fnref:assumenosideeffects-field-bug"><a href="#fn:assumenosideeffects-field-bug" class="footnote">2</a></sup></p>
<p>This example is obviously small and contrived. But does anything come to mind as a real-world use case for eliminating impossible <code class="highlighter-rouge">if</code> branches by telling R8 the range of an integer field?</p>
<h3 id="buildversionsdk_int"><code class="highlighter-rouge">Build.VERSION.SDK_INT</code></h3>
<p>As Android developers, we’re accustomed to varying implementation based on the version of the OS that our libraries and applications are running on. This is done by checking the <code class="highlighter-rouge">Build.VERSION.SDK_INT</code> integer field against known API levels.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="o">(</span><span class="nc">Build</span><span class="o">.</span><span class="na">VERSION</span><span class="o">.</span><span class="na">SDK_INT</span> <span class="o">>=</span> <span class="mi">21</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"21+ :-D"</span><span class="o">);</span>
<span class="o">}</span> <span class="k">else</span> <span class="k">if</span> <span class="o">(</span><span class="nc">Build</span><span class="o">.</span><span class="na">VERSION</span><span class="o">.</span><span class="na">SDK_INT</span> <span class="o">>=</span> <span class="mi">16</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"16+ :-)"</span><span class="o">)</span>
<span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"Pre-16 :-("</span><span class="o">);</span>
<span class="o">}</span>
</code></pre></div></div>
<p>With <code class="highlighter-rouge">-assumevalues</code>, R8 can now be used to eliminate these unused branches by specifying the supported API range.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-assumevalues class android.os.Build$VERSION {
int SDK_INT return 21..2147483647;
}
</code></pre></div></div>
<p>The range from this rule is used to see if any conditionals can be made constant.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="o">(</span><span class="nc">Build</span><span class="o">.</span><span class="na">VERSION</span><span class="o">.</span><span class="na">SDK_INT</span><span class="cm">/* [21..2147483647] */</span> <span class="o">>=</span> <span class="mi">21</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"21+ :-D"</span><span class="o">);</span>
<span class="o">}</span> <span class="k">else</span> <span class="k">if</span> <span class="o">(</span><span class="nc">Build</span><span class="o">.</span><span class="na">VERSION</span><span class="o">.</span><span class="na">SDK_INT</span><span class="cm">/* [21..2147483647] */</span> <span class="o">>=</span> <span class="mi">16</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"16+ :-)"</span><span class="o">)</span>
<span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"Pre-16 :-("</span><span class="o">);</span>
<span class="o">}</span>
</code></pre></div></div>
<p>In this example, based on the supplied range, both conditional checks will always evaluate to true.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="o">(</span><span class="kc">true</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"21+ :-D"</span><span class="o">);</span>
<span class="o">}</span> <span class="k">else</span> <span class="k">if</span> <span class="o">(</span><span class="kc">true</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"16+ :-)"</span><span class="o">)</span>
<span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"Pre-16 :-("</span><span class="o">);</span>
<span class="o">}</span>
</code></pre></div></div>
<p>With the first branch guaranteed to always be taken, dead-code elimination kicks in to remove the <code class="highlighter-rouge">else if</code> and <code class="highlighter-rouge">else</code> branches leaving only a single print.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"21+ :-D"</span><span class="o">);</span>
</code></pre></div></div>
<p>For <code class="highlighter-rouge">SDK_INT</code> conditionals in the application code we write day-to-day, there aren’t going to be branches for API levels lower than our minimum SDK version. Android’s lint tool will actually validate this with its <code class="highlighter-rouge">ObsoleteSdkInt</code> check (which you should set to error!).</p>
<p>These conditionals are far more pervasive in libraries since they tend to support a larger API range than the consuming application. It’s almost guaranteed then that the libraries have branches which will never be executed in the context of your application.</p>
<h4 id="androidx-core">AndroidX Core</h4>
<p>Whether you know it or not, these <code class="highlighter-rouge">SDK_INT</code> conditionals are all over your app. The AndroidX ‘core’ library (formerly the Support ‘compat’ library) is present in practically in 100% of apps and it exists almost exclusively to host compatibility APIs which use <code class="highlighter-rouge">SDK_INT</code> checks to vary their implementation. Its minimum supported SDK is 14 which is almost certainly lower than that of your app.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// ViewCompat.java</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">boolean</span> <span class="nf">hasOnClickListeners</span><span class="o">(</span><span class="nd">@NonNull</span> <span class="nc">View</span> <span class="n">view</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span><span class="nc">Build</span><span class="o">.</span><span class="na">VERSION</span><span class="o">.</span><span class="na">SDK_INT</span> <span class="o">>=</span> <span class="mi">15</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">view</span><span class="o">.</span><span class="na">hasOnClickListeners</span><span class="o">();</span>
<span class="o">}</span>
<span class="k">return</span> <span class="kc">false</span><span class="o">;</span>
<span class="o">}</span>
</code></pre></div></div>
<p>There are conditionals for <em>every</em> API level regardless of whether they’re needed for your app. The above example has a trivial fallback, but some of the compatibility implementations start to require quite a bit of code.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// ViewCompat.java</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">int</span> <span class="nf">getMinimumWidth</span><span class="o">(</span><span class="nd">@NonNull</span> <span class="nc">View</span> <span class="n">view</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span><span class="nc">Build</span><span class="o">.</span><span class="na">VERSION</span><span class="o">.</span><span class="na">SDK_INT</span> <span class="o">>=</span> <span class="mi">16</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">view</span><span class="o">.</span><span class="na">getMinimumWidth</span><span class="o">();</span>
<span class="o">}</span>
<span class="k">if</span> <span class="o">(!</span><span class="n">sMinWidthFieldFetched</span><span class="o">)</span> <span class="o">{</span>
<span class="k">try</span> <span class="o">{</span>
<span class="n">sMinWidthField</span> <span class="o">=</span> <span class="nc">View</span><span class="o">.</span><span class="na">class</span><span class="o">.</span><span class="na">getDeclaredField</span><span class="o">(</span><span class="s">"mMinWidth"</span><span class="o">);</span>
<span class="n">sMinWidthField</span><span class="o">.</span><span class="na">setAccessible</span><span class="o">(</span><span class="kc">true</span><span class="o">);</span>
<span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">NoSuchFieldException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span> <span class="o">}</span>
<span class="n">sMinWidthFieldFetched</span> <span class="o">=</span> <span class="kc">true</span><span class="o">;</span>
<span class="o">}</span>
<span class="k">if</span> <span class="o">(</span><span class="n">sMinWidthField</span> <span class="o">!=</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
<span class="k">try</span> <span class="o">{</span>
<span class="k">return</span> <span class="o">(</span><span class="kt">int</span><span class="o">)</span> <span class="n">sMinWidthField</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">view</span><span class="o">);</span>
<span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">Exception</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span> <span class="o">}</span>
<span class="o">}</span>
<span class="k">return</span> <span class="mi">0</span><span class="o">;</span>
<span class="o">}</span>
</code></pre></div></div>
<p>That legacy implementation after the first <code class="highlighter-rouge">if</code> sits in your APK despite few (if any) apps actually needing a pre-API 16 implementation. Some of the compatibility implementations also require entire classes for support.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// DrawableCompat.java</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="nc">Drawable</span> <span class="nf">wrap</span><span class="o">(</span><span class="nd">@NonNull</span> <span class="nc">Drawable</span> <span class="n">drawable</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span><span class="nc">Build</span><span class="o">.</span><span class="na">VERSION</span><span class="o">.</span><span class="na">SDK_INT</span> <span class="o">>=</span> <span class="mi">23</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">drawable</span><span class="o">;</span>
<span class="o">}</span> <span class="k">else</span> <span class="k">if</span> <span class="o">(</span><span class="nc">Build</span><span class="o">.</span><span class="na">VERSION</span><span class="o">.</span><span class="na">SDK_INT</span> <span class="o">>=</span> <span class="mi">21</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(!(</span><span class="n">drawable</span> <span class="k">instanceof</span> <span class="nc">TintAwareDrawable</span><span class="o">))</span> <span class="o">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="nf">WrappedDrawableApi21</span><span class="o">(</span><span class="n">drawable</span><span class="o">);</span>
<span class="o">}</span>
<span class="k">return</span> <span class="n">drawable</span><span class="o">;</span>
<span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(!(</span><span class="n">drawable</span> <span class="k">instanceof</span> <span class="nc">TintAwareDrawable</span><span class="o">))</span> <span class="o">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="nf">WrappedDrawableApi14</span><span class="o">(</span><span class="n">drawable</span><span class="o">);</span>
<span class="o">}</span>
<span class="k">return</span> <span class="n">drawable</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>If your minimum SDK is less than 23 then the <code class="highlighter-rouge">WrappedDrawableApi21</code> class is in your APK. And if your minimum SDK is less than 21 the <code class="highlighter-rouge">WrappedDrawableApi14</code> is also in your APK.</p>
<p><strong>There are over 850 <code class="highlighter-rouge">SDK_INT</code> checks in AndroidX ‘core’</strong> across every API level–double that number across all of AndroidX. You might use a few of these static helpers in your app, but it’s other libraries who using are the biggest users of these APIs. Things like <code class="highlighter-rouge">RecyclerView</code>, fragments, <code class="highlighter-rouge">CoordinatorLayout</code>, and AppCompat all support API 14 as well and so they frequently call into these methods.</p>
<p>Using <code class="highlighter-rouge">-assumevalues</code> allows R8 to eliminate compatibility implementations in these methods which will never be used by your app. This means less classes, less methods, less fields, and less code in your release APK.</p>
<h4 id="zero-overhead-abstraction">Zero-Overhead Abstraction</h4>
<p>A common theme of these posts is multiple features of R8 combining to produce really impressive results. This post is no different! The <code class="highlighter-rouge">SDK_INT</code> checks in the AndroidX ‘core’ library delegate to the framework mechanism when available. If your minimum SDK is high enough, R8 will eliminate all of the conditionals in a compat method leaving only the call to the framework.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">android.os.Build</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">android.view.View</span><span class="o">;</span>
<span class="kd">class</span> <span class="nc">ZeroOverhead</span> <span class="o">{</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">...</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">View</span> <span class="n">view</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">View</span><span class="o">(</span><span class="kc">null</span><span class="o">);</span>
<span class="n">setElevation</span><span class="o">(</span><span class="n">view</span><span class="o">,</span> <span class="mi">8</span><span class="n">f</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">setElevation</span><span class="o">(</span><span class="nc">View</span> <span class="n">view</span><span class="o">,</span> <span class="kt">float</span> <span class="n">elevation</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span><span class="nc">Build</span><span class="o">.</span><span class="na">VERSION</span><span class="o">.</span><span class="na">SDK_INT</span> <span class="o">>=</span> <span class="mi">21</span><span class="o">)</span> <span class="o">{</span>
<span class="n">view</span><span class="o">.</span><span class="na">setElevation</span><span class="o">(</span><span class="n">elevation</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>An app with a minimum SDK of 21 using <code class="highlighter-rouge">-assumevalues</code> should expect to see the <code class="highlighter-rouge">setElevation</code> static method become a simple trampoline to the built-in method.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ javac *.java
$ cat rules.txt
-keepclasseswithmembers class * {
public static void main(java.lang.String[]);
}
-dontobfuscate
-assumevalues class android.os.Build$VERSION {
int SDK_INT return 21..2147483647;
}
$ java -jar r8.jar \
--lib $ANDROID_HOME/platforms/android-28/android.jar \
--release \
--output . \
--pg-conf rules.txt \
*.class
$ $ANDROID_HOME/build-tools/28.0.3/dexdump -d classes.dex
[00013c] ZeroOverhead.main:([Ljava/lang/String;)V
0000: new-instance v1, Landroid/view/View;
0002: const/4 v0, #int 0
0003: invoke-direct {v1, v0}, Landroid/view/View;.<init>:(Landroid/content/Context;)V
0006: sget v0, Landroid/os/Build$VERSION;.SDK_INT:I
0008: const/high16 v0, #int 1090519040
000a: invoke-virtual {v1, v0}, Landroid/view/View;.setElevation:(F)V
000d: return-void
</code></pre></div></div>
<p>After running this through R8, the static <code class="highlighter-rouge">setElevation</code> method has completely disappeared. At the call site in <code class="highlighter-rouge">main</code>, bytecode index 000a now shows a direct call to the real <code class="highlighter-rouge">View.setElevation</code> method.</p>
<p>After <code class="highlighter-rouge">-assumevalues</code> removed the conditional, the body of the static <code class="highlighter-rouge">setElevation</code> method is small enough that it becomes eligible for inlining. All calls to <code class="highlighter-rouge">ViewCompat.setElevation</code> will be rewritten to directly call <code class="highlighter-rouge">view.setElevation</code>. The small penalty that would otherwise be incurred from the extra method call and conditional can be completely eliminated when they no longer serve a purpose.</p>
<h4 id="no-configuration-necessary">No Configuration Necessary</h4>
<p>If you read <a href="/avoiding-vendor-and-version-specific-vm-bugs/">the post on VM-specific workarounds</a> you might remember that D8 and R8 have a <code class="highlighter-rouge">--min-api</code> flag. When the Android Gradle plugin (AGP) invokes D8 or R8 it sets this flag to the minimum SDK version that your app supports. Starting with R8 1.4.22 which is part of AGP 3.4 beta 1 (and newer), a rule for <code class="highlighter-rouge">Build.VERSION.SDK_INT</code> is automatically added based on the <code class="highlighter-rouge">--min-api</code> flag’s value.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-assumevalues public class android.os.Build$VERSION {
public static int SDK_INT return <minApi>..2147483647;
}
</code></pre></div></div>
<p>Instead of having to know about this R8 feature and manually enable it with your minimum SDK version, the tool enables it by default so that everyone gets smaller APKs and better runtime performance.</p>
<p>Because of the use of <code class="highlighter-rouge">-assumevalues</code> for this automatic rule, the read of the <code class="highlighter-rouge">Build.VERSION.SDK_INT</code> field will be retained. You can see this in the bytecode above at index 0006. Unfortunately, switching to <code class="highlighter-rouge">-assumenosideeffects</code> won’t cause the read to be removed like an application field would. Follow <a href="https://issuetracker.google.com/issues/111763015">issuetracker.google.com/issues/111763015</a> for supporting this behavior on framework fields.</p>
<hr />
<p>Defining a range for <code class="highlighter-rouge">SDK_INT</code> is by far the most compelling demo of value assumption and now that it’s enabled by default should have a positive impact on APKs. Marking <code class="highlighter-rouge">View.isInEditMode()</code> as always false is potentially another useful default, but <a href="https://issuetracker.google.com/issues/111763015">issuetracker.google.com/issues/111763015</a> prevents it from working correctly. Other examples will likely vary from app-to-app or depend on the libraries in use.</p>
<p>The next post in the series will take a look at a few optimizations that R8 applies to values which are constants.</p>
<p><em>(This post was adapted from a part of my <a href="/digging-into-d8-and-r8">Digging into D8 and R8</a> talk. Watch the video and look out for future blog posts for more content like this.)</em></p>
<div class="footnotes">
<ol>
<li id="fn:proguard-assumevalues">
<p>Since the original presentation, ProGuard has opted to include this flag and functionality in its 6.1.0 version (currently in beta at the time of writing). So while it originated in R8, it is technically no longer R8-specific. <a href="#fnref:proguard-assumevalues" class="reversefootnote">↩</a></p>
</li>
<li id="fn:assumenosideeffects-field-bug">
<p>Currently this only works for methods and not fields. Follow <a href="https://issuetracker.google.com/issues/123080377">issuetracker.google.com/issues/123080377</a> for updates on supporting fields. <a href="#fnref:assumenosideeffects-field-bug" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
R8 Optimization: Null Data Flow Analysis (Part 2)2019-01-15T00:00:00+00:00https://jakewharton.com/r8-optimization-null-data-flow-analysis-part-2<blockquote>
<p>Note: This post is part of a series on D8 and R8, Android’s new dexer and optimizer, respectively. For an intro to D8 read <a href="/androids-java-8-support/">“Android’s Java 8 support”</a>. For an intro to R8 read <a href="/r8-optimization-staticization/">“R8 Optimization: Staticization”</a>.</p>
</blockquote>
<p><a href="/r8-optimization-null-data-flow-analysis-part-1/">Part 1 of this post</a> demonstrated R8’s ability to eliminate null checks after method inlining. This was accomplished by virtue of nullability information being present in R8’s (and D8’s) intermediate representation (IR). When the arguments flowing into a method were always non-null or always null, the now-inlined null check can be computed at compile-time.</p>
<p>Examples in the last two posts have mostly used Kotlin. To improve readability of their bytecode, I’ve been removing a section of it. The last post started with an example of a <code class="highlighter-rouge">coalesce</code> function being called from a <code class="highlighter-rouge">main</code> function.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fun</span> <span class="p"><</span><span class="nc">T</span> <span class="p">:</span> <span class="nc">Any</span><span class="p">></span> <span class="nf">coalesce</span><span class="p">(</span><span class="n">a</span><span class="p">:</span> <span class="nc">T</span><span class="p">?,</span> <span class="n">b</span><span class="p">:</span> <span class="nc">T</span><span class="p">?):</span> <span class="nc">T</span><span class="p">?</span> <span class="p">=</span> <span class="n">a</span> <span class="o">?:</span> <span class="n">b</span>
<span class="k">fun</span> <span class="nf">main</span><span class="p">(</span><span class="n">args</span><span class="p">:</span> <span class="nc">Array</span><span class="p"><</span><span class="nc">String</span><span class="p">>)</span> <span class="p">{</span>
<span class="nf">println</span><span class="p">(</span><span class="nf">coalesce</span><span class="p">(</span><span class="s">"one"</span><span class="p">,</span> <span class="s">"two"</span><span class="p">))</span>
<span class="nf">println</span><span class="p">(</span><span class="nf">coalesce</span><span class="p">(</span><span class="k">null</span><span class="p">,</span> <span class="s">"two"</span><span class="p">))</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Multiple versions of the compiled bytecode of this function were shown in that post and they all started with <code class="highlighter-rouge">sget-object v1, Ljava/lang/System;.out:Ljava/io/PrintStream;</code>. This is the bytecode looking up the static <code class="highlighter-rouge">System.out</code> field on which it can eventually invoke the <code class="highlighter-rouge">println</code> method.</p>
<p>If you compile, dex, and dump the bytecode of the Kotlin source above, however, the first bytecodes are something quite different.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ kotlinc *.kt
$ java -jar d8.jar \
--lib $ANDROID_HOME/platforms/android-28/android.jar \
--release \
--output . \
*.class kotlin-stdlib-1.3.11.jar
$ $ANDROID_HOME/build-tools/28.0.3/dexdump -d classes.dex
[00023c] NullsKt.main:([Ljava/lang/String;)V
0000: const-string v0, "args"
0002: invoke-static {v2, v0}, Lkotlin/jvm/internal/Intrinsics;.checkParameterIsNotNull:(Ljava/lang/Object;Ljava/lang/String;)V
0005: sget-object v1, Ljava/lang/System;.out:Ljava/io/PrintStream;
…
</code></pre></div></div>
<p>Instead of bytecodes representing the body of the function we wrote, the Kotlin compiler first emits a call to the standard library’s <code class="highlighter-rouge">Intrinstrics.checkParameterIsNotNull</code> function. This call is a behind-the-scenes runtime validation of a compile-time constraint.</p>
<p>Kotlin’s type system models the nullability of references. By making the parameter of my <code class="highlighter-rouge">main</code> function <code class="highlighter-rouge">Array<String></code>, I have declared it as never being null. But since this is a public API that anyone in any language can invoke, nothing prevents a non-Kotlin caller from passing null. In order to validate the non-null constraint and protect its users, the Kotlin compiler inserts defensive checks for non-null parameters in every public API function.</p>
<p><em>(Note: While there is a way to disable generation of these defensive checks, it’s not wise to do.)</em></p>
<p>Let’s take a look at how using R8 on the same source file changes the output.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cat rules.txt
-keepclasseswithmembers class * {
public static void main(java.lang.String[]);
}
-dontobfuscate
$ java -jar r8.jar \
--lib $ANDROID_HOME/platforms/android-28/android.jar \
--release \
--output . \
--pg-conf rules.txt \
*.class kotlin-stdlib-1.3.11.jar
$ $ANDROID_HOME/build-tools/28.0.3/dexdump -d classes.dex
[000314] NullsKt.main:([Ljava/lang/String;)V
0000: if-eqz v1, 0011
0002: sget-object v1, Ljava/lang/System;.out:Ljava/io/PrintStream;
…
0010: return-void
0011: const-string v1, "args"
0013: invoke-static {v1}, Lkotlin/jvm/internal/Intrinsics;.throwParameterIsNullException:(Ljava/lang/String;)V
</code></pre></div></div>
<p>The string constant load and <code class="highlighter-rouge">Intrinsics</code> method call which started the method body in the D8 output above has been replaced with a standard null check via the <code class="highlighter-rouge">if-eqz</code> bytecode. If that null check succeeds (i.e., the reference is null), the program jumps ahead to the end of the method’s bytecodes where the code that builds and throws the exception lives. As a result, in the normal operation of this method where <code class="highlighter-rouge">args</code> is non-null, the runtime can execute bytecodes index 0000 through 0010 without a jump.</p>
<p>If we make a quick conjecture about <em>why</em> the bytecode looks like this with R8 we might say that it’s the result of inlining. In part 1 we saw the <code class="highlighter-rouge">coalesce</code> function be inlined and so here the <code class="highlighter-rouge">Instrinsics.checkParameterIsNotNull</code> implementation could have just been inlined too. A quick glance at <a href="https://github.com/JetBrains/kotlin/blob/d5ebe2e66a974c3a51d6136d8e6980708cbdd058/libraries/stdlib/jvm/runtime/kotlin/jvm/internal/Intrinsics.java#L114-L118">its implementation</a> does show a standard null check and a call to <code class="highlighter-rouge">Instrincs.throwParameterIsNullException</code>.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">checkParameterIsNotNull</span><span class="o">(</span><span class="nc">Object</span> <span class="n">value</span><span class="o">,</span> <span class="nc">String</span> <span class="n">paramName</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span><span class="n">value</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
<span class="n">throwParameterIsNullException</span><span class="o">(</span><span class="n">paramName</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>But the actual R8 bytecode doesn’t match what you would expect when thinking about how inlining works. If this method was inlined, the body of the <code class="highlighter-rouge">if</code> should appear at the top of the method immediately after the check. Beyond that, despite being a tiny method it’s actually larger than R8’s inlining threshold. The only way it would be inlined is if it was used infrequently (which it isn’t). There’s a few tricks at play here which produce the actual result we’re seeing.</p>
<p>The first trick is that R8 will increase the inlining threshold for a method when it null checks an argument whose value is also an argument at the call site. Since <code class="highlighter-rouge">checkParameterIsNotNull</code> is <em>only</em> used for arguments at call sites the inlining threshold for this method goes up. The body of the method is otherwise empty so it becomes eligible and is inlined.</p>
<p>The second trick is that R8 can recognize the sequence of bytecodes which both perform a null check on an argument and then throw an exception. When this pattern is recognized, R8 assumes it’s the uncommon path for method execution. In order to optimize for the common path, the null check is inverted so that the non-null case immediately follows the check. The exception-throwing code is pushed to the bottom of the method.</p>
<p>But the <code class="highlighter-rouge">if</code> check of <code class="highlighter-rouge">checkParameterIsNotNull</code> does not match the sequence of bytecodes R8 needs to recognize the argument-check pattern. The body of the <code class="highlighter-rouge">if</code> contains a static method call instead of an exception being thrown. So the final trick is that R8 has an intrinsic which recognizes calls to <code class="highlighter-rouge">Intrinsics.throwParameterIsNullException</code> as being equivalent to throwing an exception. This allows the body to correctly match the pattern.</p>
<p>These three tricks combine to explain why R8 produces the bytecode we see above.</p>
<p>And remember, <em>every</em> method which is potentially visible to a non-Kotlin caller has this code for <em>every</em> non-null parameter. That’s a large amount of occurrences in any non-trivial app!</p>
<p>With R8 replacing a static method call with a standard null-check and moving the uncommon case to the end of the method the code retains the safety of the check while minimizing the performance implications.</p>
<h3 id="combining-null-information">Combining Null Information</h3>
<p>Once again, <a href="/r8-optimization-null-data-flow-analysis-part-1/">part 1 of this post</a> showed R8 using nullability information of values to eliminate unnecessary null checks. The first half of this part showed R8 raise inlining thresholds to ignore null checks and replace Kotlin’s <code class="highlighter-rouge">Intrinsic</code> method with standard null check bytecodes. These two features sound like they could combine to make some impactful changes. And they do!</p>
<p>This example adds a function, <code class="highlighter-rouge">String.double</code>, which just duplicates the string its called on. This function is invoked on the result of <code class="highlighter-rouge">coalesce</code> with a safe-call operator since null might be returned.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fun</span> <span class="nc">String</span><span class="p">.</span><span class="nf">double</span><span class="p">():</span> <span class="nc">String</span> <span class="p">=</span> <span class="k">this</span> <span class="p">+</span> <span class="k">this</span>
<span class="k">fun</span> <span class="nf">coalesce</span><span class="p">(</span><span class="n">a</span><span class="p">:</span> <span class="nc">String</span><span class="p">?,</span> <span class="n">b</span><span class="p">:</span> <span class="nc">String</span><span class="p">?):</span> <span class="nc">String</span><span class="p">?</span> <span class="p">=</span> <span class="n">a</span> <span class="o">?:</span> <span class="n">b</span>
<span class="k">fun</span> <span class="nf">main</span><span class="p">(</span><span class="n">args</span><span class="p">:</span> <span class="nc">Array</span><span class="p"><</span><span class="nc">String</span><span class="p">>)</span> <span class="p">{</span>
<span class="nf">println</span><span class="p">(</span><span class="nf">coalesce</span><span class="p">(</span><span class="k">null</span><span class="p">,</span> <span class="s">"two"</span><span class="p">)</span><span class="o">?.</span><span class="nf">double</span><span class="p">())</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Before looking at the R8 output, let’s enumerate the null checks which are present before dexing and optimizing:</p>
<ol>
<li><code class="highlighter-rouge">args</code> argument is checked because it’s a public function.</li>
<li>The return value of <code class="highlighter-rouge">coalesce</code> is checked before conditionally invoking <code class="highlighter-rouge">double</code>.</li>
<li><code class="highlighter-rouge">coalesce</code> checks the <code class="highlighter-rouge">first</code> argument before conditionally returning either <code class="highlighter-rouge">first</code> or <code class="highlighter-rouge">second</code>.</li>
<li><code class="highlighter-rouge">double</code>’s receiver is checked because it’s a public function.</li>
</ol>
<p>You can run D8 on the example to confirm. But running with R8 produces a pretty picture.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[000310] NullsKt.main:([Ljava/lang/String;)V
0000: if-eqz v1, 0019
0002: const-string v1, "two"
0004: new-instance v0, Ljava/lang/StringBuilder;
0006: invoke-direct {v0}, Ljava/lang/StringBuilder;.<init>:()V
0009: invoke-virtual {v0, v1}, Ljava/lang/StringBuilder;.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
000c: invoke-virtual {v0, v1}, Ljava/lang/StringBuilder;.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
000f: invoke-virtual {v0}, Ljava/lang/StringBuilder;.toString:()Ljava/lang/String;
0012: move-result-object v1
0013: sget-object v0, Ljava/lang/System;.out:Ljava/io/PrintStream;
0015: invoke-virtual {v0, v1}, Ljava/io/PrintStream;.println:(Ljava/lang/Object;)V
0018: return-void
0019: const-string v1, "args"
001b: invoke-static {v1}, Lkotlin/jvm/internal/Intrinsics;.throwParameterIsNullException:(Ljava/lang/String;)V
</code></pre></div></div>
<p>All of the null checks except the one guarding the argument were eliminated!</p>
<p>Because R8 can prove <code class="highlighter-rouge">coalesce</code> returns a non-null reference, all downstream null checks can be eliminated. This means the safe-call isn’t needed and is replaced with a normal method call. The null check on the receiver of the <code class="highlighter-rouge">double</code> function is also eliminated.</p>
<h3 id="no-inlining-required">No Inlining Required</h3>
<p>The examples so far have included inlining to aid in reducing the output. In practice, inlining won’t happen to the degree that it does in these small examples. That doesn’t prevent elimination of all null checks.</p>
<p>While I find the Kotlin examples particularly compelling here because of the forced, defensive null checks, looking at the optimization for Java is interesting because of the opposite behavior. Java doesn’t put defensive null checks on public method arguments and so data flow analysis can use other patterns for nullability signals even without inlining.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">final</span> <span class="kd">class</span> <span class="nc">Nulls</span> <span class="o">{</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">first</span><span class="o">(</span><span class="n">args</span><span class="o">));</span>
<span class="k">if</span> <span class="o">(</span><span class="n">args</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"null!"</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="nc">String</span> <span class="nf">first</span><span class="o">(</span><span class="nc">String</span><span class="o">[]</span> <span class="n">values</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span><span class="n">values</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span> <span class="k">throw</span> <span class="k">new</span> <span class="nc">NullPointerException</span><span class="o">(</span><span class="s">"values == null"</span><span class="o">);</span>
<span class="k">return</span> <span class="n">values</span><span class="o">[</span><span class="mi">0</span><span class="o">];</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Every reference is potentially-nullable in Java. As a result, it’s not uncommon to see defensive checks in library methods like <code class="highlighter-rouge">first</code> (even when annotated <code class="highlighter-rouge">@NonNull</code>!). Library methods might be large or called from all over your application and so they usually aren’t inlined. To simulate this, we can explicitly tell R8 to keep <code class="highlighter-rouge">first</code> as a method in the <code class="highlighter-rouge">rules.txt</code>.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> -keepclasseswithmembers class * {
public static void main(java.lang.String[]);
}
-dontobfuscate
<span class="gi">+-keep class Nulls {
+ public static java.lang.String first(java.lang.String[]);
+}
</span></code></pre></div></div>
<p>Even without inlining the output is favorable.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[000144] Nulls.first:([Ljava/lang/String;)Ljava/lang/String;
0000: if-eqz v1, 0006
0002: const/4 v0, #int 0
0003: aget-object v1, v1, v0
0005: return-object v1
0006: new-instance v1, Ljava/lang/NullPointerException;
0008: const-string v0, "values == null"
000a: invoke-direct {v1, v0}, Ljava/lang/NullPointerException;.<init>:(Ljava/lang/String;)V
000d: throw v1
[000170] Nulls.main:([Ljava/lang/String;)V
0000: sget-object v0, Ljava/lang/System;.out:Ljava/io/PrintStream;
0002: invoke-static {v1}, LNulls;.first:([Ljava/lang/String;)Ljava/lang/String;
0005: move-result-object v1
0006: invoke-virtual {v0, v1}, Ljava/io/PrintStream;.println:(Ljava/lang/String;)V
0009: return-void
</code></pre></div></div>
<p>In <code class="highlighter-rouge">first</code>, R8 has once again inverted the null check so that the uncommon case of throwing an exception is at the bottom of the method at index 0006. Normal execution of this method will flow from index 0000 straight down to 0005 and return.</p>
<p>In <code class="highlighter-rouge">main</code>, the explicit null check of <code class="highlighter-rouge">args</code> and its printing has disappeared. This is because R8 has tracked that the <code class="highlighter-rouge">args</code> reference flowed into <code class="highlighter-rouge">first</code> where it became impossible to be null after that call. As a result, any null checks that occur after the call to <code class="highlighter-rouge">first</code> don’t need to occur.</p>
<hr />
<p>All of these examples are small and somewhat contrived, but they demonstrate a part of the data-flow analysis that R8 is doing with regard to nullability and null checking. In the scope of your whole application whether it’s Java, Kotlin, or mixed, unnecessary null checks and unused branches can be eliminated without sacrificing the safety they otherwise afford.</p>
<p>Next week’s R8 post will cover my favorite feature of the tool. It’s also the one which I think produces the best demo and which resonates with every Android developer. Stay tuned!</p>
<p><em>(This post was adapted from a part of my <a href="/digging-into-d8-and-r8">Digging into D8 and R8</a> talk. Watch the video and look out for future blog posts for more content like this.)</em></p>
Inline Classes Make Great Database IDs2019-01-10T00:00:00+00:00https://jakewharton.com/inline-classes-make-great-database-ids<p>Kotlin 1.3’s experimental <a href="https://kotlinlang.org/docs/reference/inline-classes.html">inline class</a> feature allows creating type-safe, semantic wrappers around values which are erased at runtime. Database IDs are a perfect use case for this functionality. Combined with <a href="https://github.com/square/sqldelight">SQLDelight</a> which automatically generates model objects and APIs for querying, different table’s IDs become different types which prevent erroneous use.</p>
<p>In modeling an app that sends payments, the domain includes customers, instruments (like debit cards and bank accounts), and payments. These otherwise would all have their IDs represented by a <code class="highlighter-rouge">Long</code> allowing programming bugs such as passing a payment ID as a customer ID to go undetected.</p>
<p>Instead, define an <code class="highlighter-rouge">inline class</code> for each ID around a <code class="highlighter-rouge">Long</code> (or whatever your ID type is).</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">package</span> <span class="nn">com.example.db</span>
<span class="k">inline</span> <span class="kd">class</span> <span class="nc">CustomerId</span><span class="p">(</span><span class="kd">val</span> <span class="py">value</span><span class="p">:</span> <span class="nc">Long</span><span class="p">)</span>
<span class="k">inline</span> <span class="kd">class</span> <span class="nc">InstrumentId</span><span class="p">(</span><span class="kd">val</span> <span class="py">value</span><span class="p">:</span> <span class="nc">Long</span><span class="p">)</span>
<span class="k">inline</span> <span class="kd">class</span> <span class="nc">PaymentId</span><span class="p">(</span><span class="kd">val</span> <span class="py">value</span><span class="p">:</span> <span class="nc">Long</span><span class="p">)</span>
</code></pre></div></div>
<p>When defining your schema, tell SQLDelight to use these types for the ID columns.</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">-- src/main/sqldelight/com/example/db/Customer.sq</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">customer</span><span class="p">(</span>
<span class="n">id</span> <span class="nb">INTEGER</span> <span class="k">AS</span> <span class="n">CustomerId</span> <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">,</span>
<span class="c1">-- other columns…</span>
<span class="p">);</span>
<span class="c1">-- src/main/sqldelight/com/example/db/Instrument.sq</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">instrument</span><span class="p">(</span>
<span class="n">id</span> <span class="nb">INTEGER</span> <span class="k">AS</span> <span class="n">InstrumentId</span> <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">,</span>
<span class="c1">-- other columns…</span>
<span class="p">);</span>
</code></pre></div></div>
<p><em>(Just like when specifying any other Kotlin type for a column, you will need to register a <code class="highlighter-rouge">ColumnAdapter</code> for these types)</em></p>
<p>The payment table also uses these types for its own ID as well as the foreign key IDs to other tables.</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">-- src/main/sqldelight/com/example/db/Payment.sq</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">payment</span><span class="p">(</span>
<span class="n">id</span> <span class="nb">INTEGER</span> <span class="k">AS</span> <span class="n">PaymentId</span> <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">,</span>
<span class="n">sender_id</span> <span class="nb">INTEGER</span> <span class="k">AS</span> <span class="n">CustomerId</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="n">recipient_id</span> <span class="nb">INTEGER</span> <span class="k">AS</span> <span class="n">CustomerId</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="n">instrument_id</span> <span class="nb">INTEGER</span> <span class="k">AS</span> <span class="n">InstrumentId</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="c1">-- other columns…</span>
<span class="k">FOREIGN</span> <span class="k">KEY</span><span class="p">(</span><span class="n">sender_id</span><span class="p">)</span> <span class="k">REFERENCES</span> <span class="n">customer</span><span class="p">(</span><span class="n">id</span><span class="p">),</span>
<span class="k">FOREIGN</span> <span class="k">KEY</span><span class="p">(</span><span class="n">recipient_id</span><span class="p">)</span> <span class="k">REFERENCES</span> <span class="n">customer</span><span class="p">(</span><span class="n">id</span><span class="p">),</span>
<span class="k">FOREIGN</span> <span class="k">KEY</span><span class="p">(</span><span class="n">instrument_id</span><span class="p">)</span> <span class="k">REFERENCES</span> <span class="n">instrument</span><span class="p">(</span><span class="n">id</span><span class="p">)</span>
<span class="p">);</span>
</code></pre></div></div>
<p><em>(Note: SQLDelight will <a href="https://github.com/square/sqldelight/issues/1138">soon enforce</a> that these <code class="highlighter-rouge">FOREIGN KEY</code> relationships use the same type so that you can’t mix them up)</em></p>
<p>Named queries whose arguments or selected columns reference these IDs will now automatically use these types.</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">paymentsBySender</span><span class="p">:</span>
<span class="k">SELECT</span> <span class="n">id</span>
<span class="k">FROM</span> <span class="n">payment</span>
<span class="k">WHERE</span> <span class="n">sender_id</span> <span class="o">=</span> <span class="o">?</span><span class="p">;</span>
</code></pre></div></div>
<p>The generated Kotlin signature for this query accepts a <code class="highlighter-rouge">CustomerId</code> and the query returns <code class="highlighter-rouge">PaymentId</code>s as expected.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fun</span> <span class="nf">paymentsBySender</span><span class="p">(</span><span class="n">sender_id</span><span class="p">:</span> <span class="nc">CustomerId</span><span class="p">):</span> <span class="nc">Query</span><span class="p"><</span><span class="nc">PaymentId</span><span class="p">></span> <span class="p">{</span>
<span class="c1">// …</span>
<span class="p">}</span>
</code></pre></div></div>
<p>If you’re already looking at a single payment in this app, you might want to fetch all payments sent from that sender. With a reference to the current <code class="highlighter-rouge">Payment</code> object, you can invoke the named query to get the list.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">val</span> <span class="py">payment</span><span class="p">:</span> <span class="nc">Payment</span> <span class="p">=</span> <span class="c1">// …</span>
<span class="kd">val</span> <span class="py">bySender</span> <span class="p">=</span> <span class="n">queries</span><span class="p">.</span><span class="nf">paymentsBySender</span><span class="p">(</span><span class="n">payment</span><span class="p">.</span><span class="n">id</span><span class="p">).</span><span class="nf">executeAsList</span><span class="p">()</span>
</code></pre></div></div>
<p>Before using inline classes, this code would have compiled and returned an empty list at runtime because of programmer error. The <code class="highlighter-rouge">Payment</code>’s own <code class="highlighter-rouge">id</code> was erroneously supplied for the sender ID instead of the <code class="highlighter-rouge">sender_id</code>.</p>
<p>But because inline classes were used, this mistake can be caught at compile-time.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>PaymentPresenter.kt:189:43: error: type mismatch: inferred type is PaymentId but CustomerId was expected
val bySender = queries.paymentsBySender(payment.id).executeAsList()
^
</code></pre></div></div>
<p>After defining and using the inline classes in the schema once, this extra validation is effectively free because SQLDelight generates both the <code class="highlighter-rouge">Payment</code> model object and the function for the query.</p>
<p>A quick fix to pass <code class="highlighter-rouge">sender_id</code> allows the code to compile and also reflect the original intended behavior.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> val payment: Payment = // …
<span class="gd">-val bySender = queries.paymentsBySender(payment.id).executeAsList()
</span><span class="gi">+val bySender = queries.paymentsBySender(payment.sender_id).executeAsList()
</span></code></pre></div></div>
<p>When moving data around inside the database domain, the use of inline classes can prevent using semantically incorrect IDs. When combined with SQLDelight, these inline classes automatically apply to all of your models and query arguments adding an additional layer of safety to your database interaction. Enjoy!</p>
R8 Optimization: Null Data Flow Analysis (Part 1)2018-12-18T00:00:00+00:00https://jakewharton.com/r8-optimization-null-data-flow-analysis-part-1<blockquote>
<p>Note: This post is part of a series on D8 and R8, Android’s new dexer and optimizer, respectively. For an intro to D8 read <a href="/androids-java-8-support/">“Android’s Java 8 support”</a>. For an intro to R8 read <a href="/r8-optimization-staticization/">“R8 Optimization: Staticization”</a>.</p>
</blockquote>
<p>The <a href="/r8-optimization-staticization/">last post</a> in this series was the first to cover R8 and one of its optimizations. This post will cover an optimization which performs data flow analysis of nullability. Let’s dig in!</p>
<p>A <em>coalesce</em> function returns the first non-null argument that is provided. Running the following example, unsurprisingly, prints “one” and then “two”.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fun</span> <span class="p"><</span><span class="nc">T</span> <span class="p">:</span> <span class="nc">Any</span><span class="p">></span> <span class="nf">coalesce</span><span class="p">(</span><span class="n">a</span><span class="p">:</span> <span class="nc">T</span><span class="p">?,</span> <span class="n">b</span><span class="p">:</span> <span class="nc">T</span><span class="p">?):</span> <span class="nc">T</span><span class="p">?</span> <span class="p">=</span> <span class="n">a</span> <span class="o">?:</span> <span class="n">b</span>
<span class="k">fun</span> <span class="nf">main</span><span class="p">(</span><span class="k">vararg</span> <span class="n">args</span><span class="p">:</span> <span class="nc">String</span><span class="p">)</span> <span class="p">{</span>
<span class="nf">println</span><span class="p">(</span><span class="nf">coalesce</span><span class="p">(</span><span class="s">"one"</span><span class="p">,</span> <span class="s">"two"</span><span class="p">))</span>
<span class="nf">println</span><span class="p">(</span><span class="nf">coalesce</span><span class="p">(</span><span class="k">null</span><span class="p">,</span> <span class="s">"two"</span><span class="p">))</span>
<span class="p">}</span>
</code></pre></div></div>
<p>R8 and ProGuard will both perform function inlining when a function is small or if it’s only called in one place. Since <code class="highlighter-rouge">coalesce</code> is small, its body will be inlined to every call site to be equivalent to the following source.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fun</span> <span class="nf">main</span><span class="p">(</span><span class="k">vararg</span> <span class="n">args</span><span class="p">:</span> <span class="nc">String</span><span class="p">)</span> <span class="p">{</span>
<span class="nf">println</span><span class="p">(</span><span class="s">"one"</span> <span class="o">?:</span> <span class="s">"two"</span><span class="p">)</span>
<span class="nf">println</span><span class="p">(</span><span class="k">null</span> <span class="o">?:</span> <span class="s">"two"</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Were this actual source, the Kotlin compiler will determine that both of the elvis operators (<code class="highlighter-rouge">?:</code>) can be determined at compile-time. Compiling and dexing that fake source produces two calls to <code class="highlighter-rouge">println</code> with “one” and “two” and zero conditionals.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[000180] NullsKt.main:([Ljava/lang/String;)V
0000: sget-object v1, Ljava/lang/System;.out:Ljava/io/PrintStream;
0002: const-string v0, "one"
0004: invoke-virtual {v1, v0}, Ljava/io/PrintStream;.println:(Ljava/lang/Object;)V
0007: sget-object v1, Ljava/lang/System;.out:Ljava/io/PrintStream;
0009: const-string v0, "two"
000b: invoke-virtual {v1, v0}, Ljava/io/PrintStream;.println:(Ljava/lang/Object;)V
000e: return-void
</code></pre></div></div>
<p>But since the inlining occurs inside of R8 and not prior to running the Kotlin compiler, the actual Dalvik bytecode contains the conditionals.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[000144] NullsKt.main:([Ljava/lang/String;)V
0000: sget-object v1, Ljava/lang/System;.out:Ljava/io/PrintStream;
0002: const-string v0, "one"
0004: if-nez v0, 0006
0006: const-string v0, "two"
0008: invoke-virtual {v1, v0}, Ljava/io/PrintStream;.println:(Ljava/lang/Object;)V
000b: sget-object v1, Ljava/lang/System;.out:Ljava/io/PrintStream;
000d: const/4 v0, #int 0
000f: if-nez v0, 0010
0010: const-string v0, "two"
0012: invoke-virtual {v1, v0}, Ljava/io/PrintStream;.println:(Ljava/lang/Object;)V
0015: return-void
</code></pre></div></div>
<p>Note how bytecode index <code class="highlighter-rouge">0002</code> load the string “one” and then index <code class="highlighter-rouge">0004</code> performs a non-null check that will always succeed. This makes index <code class="highlighter-rouge">0006</code> which loads “two” dead code. Similarly, index <code class="highlighter-rouge">000d</code> loads 0 (which represents null) and then index <code class="highlighter-rouge">000f</code> does a non-null check that will always fail and fall through into index <code class="highlighter-rouge">0010</code>.</p>
<p>As mentioned in <a href="/r8-optimization-staticization/">the previous post</a>, R8 uses an intermediate representation (IR) for code. This IR uses <a href="https://en.wikipedia.org/wiki/Static_single_assignment_form">static single assignment form</a> (SSA) in order to facilitate certain optimizations. With SSA, R8 can determine how data flows through the program. For the value that flows into the first <code class="highlighter-rouge">println</code> after inlining its SSA graph looks a bit like the following.</p>
<p><a href="/static/post-image/r8null-ssa1.png"><img src="/static/post-image/r8null-ssa1.png" width="911" /></a></p>
<p>The foundational property of SSA is that each variable is only assigned once. This is why “two” is assigned to <code class="highlighter-rouge">y</code> instead of overwriting <code class="highlighter-rouge">x</code>. <code class="highlighter-rouge">z</code> uses a special phi function (Φ) to select between <code class="highlighter-rouge">x</code> or <code class="highlighter-rouge">y</code> based on which branch was taken. As you can see in the previous bytecode output, <code class="highlighter-rouge">x</code>, <code class="highlighter-rouge">y</code>, and <code class="highlighter-rouge">z</code> all wind up becoming register <code class="highlighter-rouge">v0</code> which does get overwritten–single assignment is only for the IR!</p>
<p>If we take the above graph and add nullability information to it, both <code class="highlighter-rouge">x</code> and <code class="highlighter-rouge">y</code> would be marked as non-nullable since they are both initialized with a constant. As a result, <code class="highlighter-rouge">z</code> would also be non-nullable. Since <code class="highlighter-rouge">w</code> is a field lookup of a reference, it is potentially nullable.</p>
<p>With <code class="highlighter-rouge">x</code> being non-nullable, R8 determines that the <code class="highlighter-rouge">if-nez</code> bytecode which checks if <code class="highlighter-rouge">x</code> is non-null will always be true and thus is useless. The false branch of the conditional which assigns <code class="highlighter-rouge">y</code> will never be taken and so it is also useless.</p>
<p><a href="/static/post-image/r8null-ssa2.png"><img src="/static/post-image/r8null-ssa2.png" width="911" /></a></p>
<p>These useless bytecodes can then be pruned from the graph since we know that they are dead code.</p>
<p><a href="/static/post-image/r8null-ssa3.png"><img src="/static/post-image/r8null-ssa3.png" width="577" /></a></p>
<p><code class="highlighter-rouge">z</code> is now a phi function on a single variable, <code class="highlighter-rouge">x</code>, which means we can just replace all usages <code class="highlighter-rouge">z</code> directly with <code class="highlighter-rouge">x</code>.</p>
<p><a href="/static/post-image/r8null-ssa4.png"><img src="/static/post-image/r8null-ssa4.png" width="403" /></a></p>
<p>What’s left is just the <code class="highlighter-rouge">System.out</code> lookup into <code class="highlighter-rouge">w</code>, assignment of the “one” constant into <code class="highlighter-rouge">x</code>, and then the call to <code class="highlighter-rouge">println</code> on <code class="highlighter-rouge">w</code> with the value <code class="highlighter-rouge">x</code>.</p>
<p>The above was only the SSA graph which flows into the first <code class="highlighter-rouge">println</code>. The second <code class="highlighter-rouge">println</code> is the inverse case where the value is initialized to null, a null check is performed, and then a fallback value is conditionally set.</p>
<p><a href="/static/post-image/r8null-ssa5.png"><img src="/static/post-image/r8null-ssa5.png" width="911" /></a></p>
<p>With the SSA IR, R8 is able to determine that both conditionals are useless after the inlining of <code class="highlighter-rouge">coalesce</code> and remove the dead branches.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ kotlinc *.kt
$ cat rules.txt
-keepclasseswithmembers class * {
public static void main(java.lang.String[]);
}
-dontobfuscate
$ java -jar r8.jar \
--lib $ANDROID_HOME/platforms/android-28/android.jar \
--release \
--output . \
--pg-conf rules.txt \
*.class kotlin-stdlib-1.3.11.jar
$ $ANDROID_HOME/build-tools/28.0.3/dexdump -d classes.dex
[000340] NullsKt.main:([Ljava/lang/String;)V
0000: sget-object v1, Ljava/lang/System;.out:Ljava/io/PrintStream;
0002: const-string v0, "one"
0004: invoke-virtual {v1, v0}, Ljava/io/PrintStream;.println:(Ljava/lang/Object;)V
0007: sget-object v1, Ljava/lang/System;.out:Ljava/io/PrintStream;
0009: const-string v0, "two"
000b: invoke-virtual {v1, v0}, Ljava/io/PrintStream;.println:(Ljava/lang/Object;)V
000e: return-void
</code></pre></div></div>
<p>The final Dalvik bytecode now matches that which the manually-inlined source file above produced.</p>
<h4 id="analysis-inside-d8">Analysis Inside D8</h4>
<p>In attempting to create the bytecode that would be generated after inlining but before nullability analysis eliminated dead code I tried to use equivalent Java.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Nulls</span> <span class="o">{</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">...</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">Object</span> <span class="n">first</span> <span class="o">=</span> <span class="s">"one"</span><span class="o">;</span>
<span class="k">if</span> <span class="o">(</span><span class="n">first</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
<span class="n">first</span> <span class="o">=</span> <span class="s">"two"</span><span class="o">;</span>
<span class="o">}</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">first</span><span class="o">);</span>
<span class="nc">Object</span> <span class="n">second</span> <span class="o">=</span> <span class="kc">null</span><span class="o">;</span>
<span class="k">if</span> <span class="o">(</span><span class="n">second</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
<span class="n">second</span> <span class="o">=</span> <span class="s">"two"</span><span class="o">;</span>
<span class="o">}</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">second</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>When you compile, dex with D8, and dump the bytecode from this example, though, the conditionals are still eliminated.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ javac *.java
$ java -jar d8.jar \
--lib $ANDROID_HOME/platforms/android-28/android.jar \
--release \
--output . \
*.class
$ $ANDROID_HOME/build-tools/28.0.3/dexdump -d classes.dex
[000224] Nulls.main:([Ljava/lang/String;)V
0000: sget-object v1, Ljava/lang/System;.out:Ljava/io/PrintStream;
0002: const-string v0, "one"
0004: invoke-virtual {v1, v0}, Ljava/io/PrintStream;.println:(Ljava/lang/Object;)V
0007: sget-object v1, Ljava/lang/System;.out:Ljava/io/PrintStream;
0009: const-string v0, "two"
000b: invoke-virtual {v1, v0}, Ljava/io/PrintStream;.println:(Ljava/lang/Object;)V
000e: return-void
</code></pre></div></div>
<p>The reason that this happens is because the same IR is used by D8 and the nullability information is still present. Even without doing any of R8 optimizations, when conditionals are present in the IR that are trivially determined to be always true or always false then dead code elimination can occur.</p>
<p>If you use the legacy <code class="highlighter-rouge">dx</code> tool whose IR does not contain this information the bytecode will retain the conditionals and dead code.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ $ANDROID_HOME/build-tools/28.0.3/dx --dex --output=classes.dex *.class
$ $ANDROID_HOME/build-tools/28.0.3/dexdump -d classes.dex
[000204] Nulls.main:([Ljava/lang/String;)V
0000: const-string v0, "one"
0002: if-nez v0, 0006
0004: const-string v0, "two"
0006: sget-object v1, Ljava/lang/System;.out:Ljava/io/PrintStream;
0008: invoke-virtual {v1, v0}, Ljava/io/PrintStream;.println:(Ljava/lang/Object;)V
000b: const/4 v0, #int 0
000c: if-nez v0, 0010
000e: const-string v0, "two"
0010: sget-object v1, Ljava/lang/System;.out:Ljava/io/PrintStream;
0012: invoke-virtual {v1, v0}, Ljava/io/PrintStream;.println:(Ljava/lang/Object;)V
0015: return-void
</code></pre></div></div>
<p>So while the data flow analysis really shines when optimizations like inlining are being applied by R8, if constant conditionals and dead code are present directly from source they’ll still be eliminated by D8.</p>
<hr />
<p>This post only scratches the surface of the data flow analysis inside R8. The next post will continue to expand on the nullability analysis with respect to how Kotlin enforces nullability constraints at runtime.</p>
<p><em>(This post was adapted from a part of my <a href="/digging-into-d8-and-r8">Digging into D8 and R8</a> talk. Watch the video and look out for future blog posts for more content like this.)</em></p>
R8 Optimization: Staticization2018-12-11T00:00:00+00:00https://jakewharton.com/r8-optimization-staticization<blockquote>
<p>Note: This post is part of a series on D8 and R8, Android’s new dexer and optimizer, respectively. For an intro to D8 read <a href="/androids-java-8-support/">“Android’s Java 8 support”</a>. This post introduces R8.</p>
</blockquote>
<p>The first three posts (<a href="/androids-java-8-support/">1</a>, <a href="/androids-java-9-10-11-and-12-support/">2</a>, <a href="/avoiding-vendor-and-version-specific-vm-bugs/">3</a>) in this series explored D8. Among its core responsibility of converting Java bytecode to Dalvik bytecode, it desugars new Java language features and works around vendor- and version-specific bugs in Android’s VMs.</p>
<p>In general, D8 doesn’t perform optimization. It may choose to use Dalvik bytecodes which more efficiently represent the intent of Java bytecodes (as seen with <a href="/avoiding-vendor-and-version-specific-vm-bugs/#not-a-not">the <code class="highlighter-rouge">not-int</code> example</a>). Or, in the process of desugaring language features, it may choose to optimize the desugared code it is generating. Aside from these very localized changes, D8 otherwise performs a direct translation.</p>
<p>R8 is a version of D8 that also performs optimization. It’s not a separate tool or codebase, just the same tool operating in a more advanced mode. Where D8 first parses Java bytecode into its own intermediate representation (IR) and then writes out the Dalvik bytecode, R8 adds optimization passes over the IR before its written out.</p>
<p>This post (and a bunch of future posts) are going to explore some of the individual optimizations that R8 performs. We start with an optimization called <em>staticization</em> which means the act of making something static.</p>
<h3 id="companion-objects">Companion Objects</h3>
<p>Kotlin uses companion objects to model the features of Java’s <code class="highlighter-rouge">static</code> modifier. They’re actually a much more powerful language feature allowing things like inheritance and implementing interfaces. That power comes with an associated cost, however, and we pay for that cost regardless of whether we’re using the added power or just emulating <code class="highlighter-rouge">static</code>.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fun</span> <span class="nf">main</span><span class="p">(</span><span class="k">vararg</span> <span class="n">args</span><span class="p">:</span> <span class="nc">String</span><span class="p">)</span> <span class="p">{</span>
<span class="nf">println</span><span class="p">(</span><span class="nc">Greeter</span><span class="p">.</span><span class="nf">hello</span><span class="p">().</span><span class="nf">greet</span><span class="p">(</span><span class="s">"Olive"</span><span class="p">))</span>
<span class="p">}</span>
<span class="kd">class</span> <span class="nc">Greeter</span><span class="p">(</span><span class="kd">val</span> <span class="py">greeting</span><span class="p">:</span> <span class="nc">String</span><span class="p">)</span> <span class="p">{</span>
<span class="k">fun</span> <span class="nf">greet</span><span class="p">(</span><span class="n">name</span><span class="p">:</span> <span class="nc">String</span><span class="p">)</span> <span class="p">=</span> <span class="s">"$greeting, $name!"</span>
<span class="k">companion</span> <span class="k">object</span> <span class="p">{</span>
<span class="k">fun</span> <span class="nf">hello</span><span class="p">()</span> <span class="p">=</span> <span class="nc">Greeter</span><span class="p">(</span><span class="s">"Hello"</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>In this example, the <code class="highlighter-rouge">Greeter</code> class uses a <code class="highlighter-rouge">companion object</code> to expose functionality that isn’t tied to instances of <code class="highlighter-rouge">Greeter</code>. A convenience factory <code class="highlighter-rouge">hello</code> returns instances of <code class="highlighter-rouge">Greeter</code> initialized with the string “Hello”. A <code class="highlighter-rouge">main</code> function calls the factory and then greets my dog Olive.</p>
<p>Compiling with <code class="highlighter-rouge">kotlinc</code>, dexing with D8, and dumping the Dalvik bytecode with <code class="highlighter-rouge">dexdump</code> we can see how this is implemented.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ kotlinc *.kt
$ java -jar d8.jar \
--lib $ANDROID_HOME/platforms/android-28/android.jar \
--release \
--output . \
*.class
$ $ANDROID_HOME/build-tools/28.0.3/dexdump -d classes.dex
…
[000370] GreeterKt.main:([Ljava/lang/String;)V
0000: sget-object v1, LGreeter;.Companion:LGreeter$Companion;
0002: invoke-virtual {v1}, LGreeter$Companion;.hello:()LGreeter;
0005: move-result-object v1
0006: const-string v0, "Olive"
0008: invoke-virtual {v1, v0}, LGreeter;.greet:(Ljava/lang/String;)Ljava/lang/String;
000b: move-result-object v1
000c: sget-object v0, Ljava/lang/System;.out:Ljava/io/PrintStream;
000e: invoke-virtual {v0, v1}, Ljava/io/PrintStream;.println:(Ljava/lang/Object;)V
0011: return-void
…
</code></pre></div></div>
<p>Bytecode index <code class="highlighter-rouge">0000</code> loads an instance of the <code class="highlighter-rouge">Greeter$Companion</code> class from a static <code class="highlighter-rouge">Companion</code> field on <code class="highlighter-rouge">Greeter</code>. Index <code class="highlighter-rouge">0002</code> then makes a virtual method call to the <code class="highlighter-rouge">hello</code> function on that instance.</p>
<p>Looking at the nested <code class="highlighter-rouge">Companion</code> class confirms that it contains virtual (aka non-static methods).</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Virtual methods -
#0 : (in LGreeter$Companion;)
name : 'hello'
type : '()LGreeter;'
access : 0x0011 (PUBLIC FINAL)
[000314] Greeter.Companion.hello:(Ljava/lang/String;)Ljava/lang/String;
0000: new-instance v0, LGreeter;
0002: const-string v1, "Hello"
0004: invoke-direct {v0, v1}, LGreeter;.<init>:(Ljava/lang/String;)V
0007: return-object v0
</code></pre></div></div>
<p>The use of a companion on <code class="highlighter-rouge">Greeter</code> means that a second, nested class named <code class="highlighter-rouge">Companion</code> is generated which adds to our binary size and slows startup because of additional class loading. The singleton instance of this class is retained in memory for the life of our application adding memory pressure. And finally, the use of instance methods require virtual calls which are slower than static calls. Granted, the impact of all these things for just one class is <em>extremely</em> minor, but in a large application written entirely in Kotlin it begins to contribute non-trivial overhead.</p>
<p>We can convert the Java classfiles to Dalvik using R8 instead of D8 and see what optimizations it applies. The flags to run R8 is nearly identical to D8 except it requires adding <code class="highlighter-rouge">--pg-conf</code> to supply a <a href="https://www.guardsquare.com/en/products/proguard/manual/usage">ProGuard-compatible configuration file</a>. The one in use here keeps the <code class="highlighter-rouge">main</code> method as an entry point (otherwise the dex file would be empty) and disables class and method name obfuscation for the sake of readability.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cat rules.txt
-keepclasseswithmembers class * {
public static void main(java.lang.String[]);
}
-dontobfuscate
$ java -jar r8.jar \
--lib $ANDROID_HOME/platforms/android-28/android.jar \
--release \
--output . \
--pg-conf rules.txt \
*.class
</code></pre></div></div>
<p>R8 will produce a <code class="highlighter-rouge">classes.dex</code> just like D8 except with contents that have been optimized.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ $ANDROID_HOME/build-tools/28.0.3/dexdump -d classes.dex
…
[000234] GreeterKt.main:([Ljava/lang/String;)V
0000: invoke-static {}, LGreeter;.hello:()LGreeter;
0003: move-result-object v1
0004: const-string v0, "Olive"
0006: invoke-virtual {v1, v0}, LGreeter;.greet:(Ljava/lang/String;)Ljava/lang/String;
0009: move-result-object v1
000a: sget-object v0, Ljava/lang/System;.out:Ljava/io/PrintStream;
000c: invoke-virtual {v0, v1}, Ljava/io/PrintStream;.println:(Ljava/lang/Object;)V
000f: return-void
…
</code></pre></div></div>
<p>The <code class="highlighter-rouge">main</code> method has changed slightly from the original version. Instead of an <code class="highlighter-rouge">sget-object</code> to look up the <code class="highlighter-rouge">Companion</code> instance and an <code class="highlighter-rouge">invoke-virtual</code> to call a <code class="highlighter-rouge">hello</code> instance method, only an <code class="highlighter-rouge">invoke-static</code> remains. It’s also important to note that R8 hasn’t just made the <code class="highlighter-rouge">hello</code> method static inside the <code class="highlighter-rouge">Companion</code> class, it has moved the method from the <code class="highlighter-rouge">Companion</code> to be directly on the <code class="highlighter-rouge">Greeter</code> class.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> #1 : (in LGreeter;)
name : 'hello'
type : '(Ljava/lang/String;)Ljava/lang/String;'
access : 0x0019 (PUBLIC STATIC FINAL)
[0002bc] Greeter.hello:(Ljava/lang/String;)Ljava/lang/String;
[000240] Greeter.hello:()LGreeter;
0000: new-instance v0, LGreeter;
0002: const-string v1, "Hello"
0004: invoke-direct {v0, v1}, LGreeter;.<init>:(Ljava/lang/String;)V
0007: return-object v0
</code></pre></div></div>
<p>With the <code class="highlighter-rouge">hello</code> method having been moved, the entire <code class="highlighter-rouge">Companion</code> class and the singleton field holding its instance on <code class="highlighter-rouge">Greeter</code> have both been removed.</p>
<p>This is staticization in practice. R8 finds occurrences of instance methods where the instance isn’t actually required and makes them static. It also has special knowledge of how Kotlin implements companions so that in addition to making their methods static the extra class they’d otherwise generate can also be removed.</p>
<h4 id="source-transformation">Source Transformation</h4>
<p>Understanding exactly how a Kotlin companion is represented in bytecode and how R8’s optimization works in bytecode can be challenging. In order to better understand both of these things we can emulate them at the source-code level.</p>
<p>The Kotlin compiler compiles the original <code class="highlighter-rouge">Greeter</code> class into Java bytecode which approximates to the following Java source code.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">final</span> <span class="kd">class</span> <span class="nc">Greeter</span> <span class="o">{</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">Companion</span> <span class="nc">Companion</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">Companion</span><span class="o">();</span>
<span class="kd">private</span> <span class="kd">final</span> <span class="nc">String</span> <span class="n">greeting</span><span class="o">;</span>
<span class="kd">public</span> <span class="nf">Greeter</span><span class="o">(</span><span class="nc">String</span> <span class="n">greeting</span><span class="o">)</span> <span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">greeting</span> <span class="o">=</span> <span class="n">greeting</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="nc">String</span> <span class="nf">getGreeting</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">greeting</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="nc">String</span> <span class="nf">greet</span><span class="o">(</span><span class="nc">String</span> <span class="n">name</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">greeting</span> <span class="o">+</span> <span class="s">", "</span> <span class="o">+</span> <span class="n">name</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="kd">class</span> <span class="nc">Companion</span> <span class="o">{</span>
<span class="kd">private</span> <span class="nf">Companion</span><span class="o">()</span> <span class="o">{}</span>
<span class="kd">public</span> <span class="nc">Greeter</span> <span class="nf">hello</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="nf">Greeter</span><span class="o">(</span><span class="s">"Hello"</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The <code class="highlighter-rouge">val greeting: String</code> primary constructor property declaration is translated into a private field, constructor parameter, constructor assignment statement, and getter method. The <code class="highlighter-rouge">companion object</code> becomes a nested class named <code class="highlighter-rouge">Companion</code> and the enclosing <code class="highlighter-rouge">Greeter</code> class keeps a static, final singleton instance of it.</p>
<p>The main method is put into yet another class called <code class="highlighter-rouge">GreeterKt</code> which is based on the filename, <code class="highlighter-rouge">Greeter.kt</code>.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">final</span> <span class="kd">class</span> <span class="nc">GreeterKt</span> <span class="o">{</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="nc">Greeter</span><span class="o">.</span><span class="na">Companion</span><span class="o">.</span><span class="na">hello</span><span class="o">().</span><span class="na">greet</span><span class="o">(</span><span class="s">"Olive"</span><span class="o">));</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>In order to access the <code class="highlighter-rouge">hello</code> factory method, the <code class="highlighter-rouge">main</code> method calls through the static <code class="highlighter-rouge">Companion</code> field.</p>
<p>R8’s optimization alters the code into what we otherwise would have written if the original <code class="highlighter-rouge">Greeter</code> was written in Java.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> public final class Greeter {
<span class="gd">- public static final Companion Companion = new Companion();
-
</span> private final String greeting;
<span class="err">@@</span>
<span class="gd">- public static final class Companion {
- private Companion() {}
-
- public Greeter hello() {
- return new Greeter("Hello");
- }
- }
</span><span class="gi">+ public static Greeter hello() {
+ return new Greeter("Hello");
+ }
</span> }
</code></pre></div></div>
<p>The <code class="highlighter-rouge">hello</code> method becomes a static method directly inside <code class="highlighter-rouge">Greeter</code> and the <code class="highlighter-rouge">Companion</code> class and singleton instance field are removed.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> public final class GreeterKt {
public static void main(String[] args) {
<span class="gd">- System.out.println(Greeter.Companion.hello().greet("Olive"));
</span><span class="gi">+ System.out.println(Greeter.hello().greet("Olive"));
</span> }
}
</code></pre></div></div>
<p>The <code class="highlighter-rouge">main</code> method is also updated to reflect this change, again looking more like if it were originally written in Java.</p>
<h4 id="jvmstatic"><code class="highlighter-rouge">@JvmStatic</code></h4>
<p>If you’re familiar with Kotlin and its Java interoperability story, using the <code class="highlighter-rouge">@JvmStatic</code> annotation might have come to mind to achieve a similar effect.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> companion object {
<span class="gi">+ @JvmStatic
</span> fun hello() = Greeter("Hello")
</code></pre></div></div>
<p>With the annotation added to the original example, running it through D8 only and dumping the bytecode shows an interesting result.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ kotlinc *.kt
$ java -jar d8.jar \
--lib $ANDROID_HOME/platforms/android-28/android.jar \
--release \
--output . \
*.class
$ $ANDROID_HOME/build-tools/28.0.3/dexdump -d classes.dex
…
#2 : (in LGreeter;)
name : 'hello'
type : '()LGreeter;'
access : 0x0019 (PUBLIC STATIC FINAL)
[00042c] Greeter.hello:()LGreeter;
0000: sget-object v0, LGreeter;.Companion:LGreeter$Companion;
0002: invoke-virtual {v0, v1}, LGreeter$Companion;.hello:()LGreeter;
0005: move-result-object v1
0006: return-object v1
…
</code></pre></div></div>
<p>A static <code class="highlighter-rouge">hello</code> method was added to the <code class="highlighter-rouge">Greeter</code> class, but it’s just a trampoline into the <code class="highlighter-rouge">Companion</code> instance and the instance method of the same name.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[000234] GreeterKt.main:([Ljava/lang/String;)V
0000: sget-object v1, LGreeter;.Companion:LGreeter$Companion;
0002: invoke-virtual {v1}, LGreeter$Companion;.hello:()LGreeter;
…
</code></pre></div></div>
<p>And even with that static method present, Kotlin callers still do the <code class="highlighter-rouge">Companion</code> instance lookup and virtual method call.</p>
<p>Even with <code class="highlighter-rouge">@JvmStatic</code> present, R8 will still perform the staticization optimization. The <code class="highlighter-rouge">Companion</code>’s <code class="highlighter-rouge">greet</code> method body will move into the static <code class="highlighter-rouge">greet</code> method on <code class="highlighter-rouge">Greeter</code>, the <code class="highlighter-rouge">main</code> function will do a static method call, and the entire <code class="highlighter-rouge">Companion</code> class will be removed.</p>
<h3 id="more-than-companions">More Than Companions</h3>
<p>This optimization isn’t limited to only Kotlin <code class="highlighter-rouge">companion object</code>s. Regular Kotlin <code class="highlighter-rouge">object</code>s will have their methods made <code class="highlighter-rouge">static</code>.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@Module</span>
<span class="kd">object</span> <span class="nc">HelloGreeterModule</span> <span class="p">{</span>
<span class="nd">@Provides</span> <span class="k">fun</span> <span class="nf">greeter</span><span class="p">()</span> <span class="p">=</span> <span class="nc">Greeter</span><span class="p">(</span><span class="s">"Hello"</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Java classes will also receive this optimization when the instance is not needed.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">final</span> <span class="kd">class</span> <span class="nc">Thing</span> <span class="o">{</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">Thing</span> <span class="no">INSTANCE</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">Thing</span><span class="o">();</span>
<span class="kd">private</span> <span class="nf">Thing</span><span class="o">()</span> <span class="o">{}</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">doThing</span><span class="o">()</span> <span class="o">{</span>
<span class="c1">// …</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Running R8 on these examples and validating the resulting bytecode is left as an exercise for the reader.</p>
<hr />
<p>In summary, staticization takes instance methods which don’t actually require access to an instance and makes them static. For Kotlin, it understands the bytecode of companion objects and can often eliminate them entirely when they’re only being used to emulate Java’s <code class="highlighter-rouge">static</code>.</p>
<p>Many R8 optimizations are aware of Kotlin-specific bytecode patterns in order to make them more effective. Stay tuned for the next post which features another R8 optimization that works well with Kotlin.</p>
<p><em>(This post was adapted from a part of my <a href="/digging-into-d8-and-r8">Digging into D8 and R8</a> talk. Watch the video and look out for future blog posts for more content like this.)</em></p>
Avoiding Vendor- and Version-Specific VM Bugs2018-12-04T00:00:00+00:00https://jakewharton.com/avoiding-vendor-and-version-specific-vm-bugs<blockquote>
<p>Note: This post is part of a series on D8 and R8, Android’s new dexer and optimizer, respectively. For an intro to D8 read <a href="/androids-java-8-support/">“Android’s Java 8 support”</a>.</p>
</blockquote>
<p>The first two posts (<a href="/androids-java-8-support/">1</a>, <a href="/androids-java-9-10-11-and-12-support/">2</a>) in this series explored how D8 is responsible for desugaring new Java language features to work on all versions of Android. Desugaring is the more interesting feature to demonstrate, but it’s secondary functionality of D8. The primary responsibility is converting the stack-based Java bytecode into register-based Dalvik bytecode so that it can run on Android’s VM.</p>
<p>At this point in Android’s tenure it’d be reasonable to think that this conversion (called <em>dexing</em>) is a solved problem. During the process of building and rolling out D8, however, interesting vendor-specific and version-specific bugs in different VMs were uncovered which this post is going to explore.</p>
<h3 id="not-a-not">Not A Not</h3>
<p>D8 takes compiled Java bytecode and produces equivalent functionality using Dalvik bytecode. We can see this with a simple example that uses Java’s bitwise not operator.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Not</span> <span class="o">{</span>
<span class="kd">static</span> <span class="kt">void</span> <span class="nf">print</span><span class="o">(</span><span class="kt">int</span> <span class="n">value</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(~</span><span class="n">value</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Compiling and dumping the class file shows the bytecodes that are used to implement this feature.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ javac *.java
$ javap -c *.class
class Not {
static void print(int);
Code:
0: getstatic #2 // Field java/lang/System.out:Ljava/io/PrintStream;
3: iload_0
4: iconst_m1
5: ixor
6: invokevirtual #3 // Method java/io/PrintStream.println:(I)V
9: return
}
</code></pre></div></div>
<p>Bytecode index <code class="highlighter-rouge">3</code>, <code class="highlighter-rouge">4</code>, and <code class="highlighter-rouge">5</code> load the argument value onto the stack, load the constant -1, and perform a bitwise exclusive-or. If your bitwise skills are a little rusty, -1 is represented as all 1s and an exclusive-or sets a bit if and only if one of the two bits is set.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>00010100 (value)
xor
11111111 (-1)
=
11101011
</code></pre></div></div>
<p>By performing an exclusive-or on a number whose bits are all set to 1, we are left with a number whose bits are the opposite of the original yielding the bitwise not.</p>
<p>Running this through D8 shows the operation is implemented similarly in Dalvik bytecode.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ java -jar d8.jar \
--lib $ANDROID_HOME/platforms/android-28/android.jar \
--release \
--output . \
*.class
$ $ANDROID_HOME/build-tools/28.0.3/dexdump -d classes.dex
[000134] Not.print:(I)V
0000: sget-object v0, Ljava/lang/System;.out:Ljava/io/PrintStream;
0002: xor-int/lit8 v1, v1, #int -1
0004: invoke-virtual {v0, v1}, Ljava/io/PrintStream;.println:(I)V
0007: return-void
</code></pre></div></div>
<p>Index <code class="highlighter-rouge">0002</code> performs an exclusive-or on register <code class="highlighter-rouge">v1</code> (the argument value) with a constant of -1 and stores it back into <code class="highlighter-rouge">v1</code>. This is a very straightforward mapping from Java bytecode and if you didn’t know any better it wouldn’t be given a second thought. But its inclusion in this post should tip you off that there is more to the story.</p>
<p>All of the Dalvik bytecodes are <a href="https://source.android.com/devices/tech/dalvik/dalvik-bytecode">available for browsing</a> on the Android developer documentation site. If you look closely, there’s a unary operator section which contains a bytecode called <code class="highlighter-rouge">not-int</code>. Instead of doing an exclusive-or on the argument value with -1 a dedicated bitwise not bytecode could be used. This has the potential for using more efficient machine instructions and hardware in the CPU. So why isn’t it being used?</p>
<p>The answer lies with the old <code class="highlighter-rouge">dx</code> tool and the fact that it also does not use the <code class="highlighter-rouge">not-int</code> instruction.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ $ANDROID_HOME/build-tools/28.0.3/dx \
--dex \
--output=classes.dex \
*.class
[000130] Not.print:(I)V
0000: sget-object v0, Ljava/lang/System;.out:Ljava/io/PrintStream;
0002: xor-int/lit8 v1, v2, #int -1
0004: invoke-virtual {v0, v1}, Ljava/io/PrintStream;.println:(I)V
0007: return-void
</code></pre></div></div>
<p>The old <code class="highlighter-rouge">dx</code> tool is hosted in <code class="highlighter-rouge">dalvik/dx/</code> of AOSP. If we grep its codebase, we can find the constant used for the <code class="highlighter-rouge">not-int</code> instruction.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ grep -r -C 1 'not-int' src/com/android/dx/io
OpcodeInfo.java-522- public static final Info NOT_INT =
OpcodeInfo.java:523: new Info(Opcodes.NOT_INT, "not-int",
OpcodeInfo.java-524- InstructionCodec.FORMAT_12X, IndexType.NONE);
</code></pre></div></div>
<p>So while <code class="highlighter-rouge">dx</code> knows that the instruction exists, when you grep its codebase for <em>uses</em> of that constant when converting from a class file there are zero! For comparison I’ve also included the <code class="highlighter-rouge">if-eq</code> bytecode’s constant.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ grep -r -C 1 'NOT_INT' src/com/android/dx/cf
$ grep -r -C 1 'IF_EQ' src/com/android/dx/cf
code/RopperMachine.java-885- case ByteOps.IFNULL: {
code/RopperMachine.java:886: return RegOps.IF_EQ;
code/RopperMachine.java-887- }
</code></pre></div></div>
<p>This means that the <code class="highlighter-rouge">dx</code> tool will never emit a <code class="highlighter-rouge">not-int</code> instruction no matter what Java bytecodes were used. This is unfortunate, but ultimately isn’t that big of a deal.</p>
<p>The real problem stems from the fact that because the bytecode was never used by the canonical dexing tool, some vendors decided that they wouldn’t bother supporting it in their Dalvik VM’s JIT! Once D8 came along and started using the full bytecode set, JIT-compiled apps running on these specific phones would crash. As a result, D8 can’t use the <code class="highlighter-rouge">not-int</code> instruction in this case even if it wants to.</p>
<p>With the introduction of the ART VM in API 21, all phones now have support for this instruction. As a result, passing <code class="highlighter-rouge">--min-api 21</code> to D8 will change the bytecodes used to leverage <code class="highlighter-rouge">not-int</code>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ java -jar d8.jar \
--lib $ANDROID_HOME/platforms/android-28/android.jar \
--release \
--min-api 21 \
--output . \
*.class
$ $ANDROID_HOME/build-tools/28.0.3/dexdump -d classes.dex
[000134] Not.print:(I)V
0000: sget-object v0, Ljava/lang/System;.out:Ljava/io/PrintStream;
0002: not-int v1, v1
0003: invoke-virtual {v0, v1}, Ljava/io/PrintStream;.println:(I)V
0006: return-void
</code></pre></div></div>
<p>Index <code class="highlighter-rouge">0002</code> now contains the more specific instruction as we expect.</p>
<p>In a similar manner to how language features are desugared to work on older versions of Android, D8 can change the shape of individual bytecodes to ensure compatibility. As the ecosystem and our minimum API level rises, D8 will automatically use the more efficient bytecodes.</p>
<h3 id="long-compare">Long Compare</h3>
<p>Even when all of the instructions in use are supported, vendor-specific JITs are software like any other and can contain bugs. This happened close to home in code that was present in OkHttp and Okio.</p>
<p>Both libraries deal in moving and counting bytes. Their methods frequently start with a check for a negative count (which is invalid) and then a zero count (no work to do).</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">LongCompare</span> <span class="o">{</span>
<span class="kd">static</span> <span class="kt">void</span> <span class="nf">somethingWithBytes</span><span class="o">(</span><span class="kt">long</span> <span class="n">byteCount</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span><span class="n">byteCount</span> <span class="o"><</span> <span class="mi">0</span><span class="o">)</span> <span class="k">throw</span> <span class="k">new</span> <span class="nc">IllegalArgumentException</span><span class="o">(</span><span class="s">"byteCount < 0"</span><span class="o">);</span>
<span class="k">if</span> <span class="o">(</span><span class="n">byteCount</span> <span class="o">==</span> <span class="mi">0</span><span class="o">)</span> <span class="k">return</span><span class="o">;</span> <span class="c1">// Nothing to do!</span>
<span class="c1">// Do something…</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>When you compile and dex this code, the constant 0 is loaded and then two comparisons are made.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ javac *.java
$ java -jar d8.jar \
--lib $ANDROID_HOME/platforms/android-28/android.jar \
--release \
--output . \
*.class
$ $ANDROID_HOME/build-tools/28.0.3/dexdump -d classes.dex
[000138] LongCompare.somethingWithBytes:(J)V
0000: const-wide/16 v0, #int 0
0002: cmp-long v2, v3, v0
0004: if-ltz v2, 000b
0006: cmp-long v2, v3, v0
0008: if-nez v2, 000a
…
</code></pre></div></div>
<p>Based on these bytecodes, we can infer that <code class="highlighter-rouge">cmp-long</code> produces a value that’s either less-than zero, zero, or greater-than zero. After each comparison, a check for less-than zero occurs and then a check for non-zero, respectively. But if a single <code class="highlighter-rouge">cmp-long</code> produces the comparison result, why does index <code class="highlighter-rouge">0006</code> perform it a second time?</p>
<p>The reason is that one vendor-specific JIT will crash if a non-zero check is performed immediately after a less-than zero check. This would cause the program to see impossible exceptions such as a <code class="highlighter-rouge">NullPointerException</code> when only dealing with <code class="highlighter-rouge">long</code>s.</p>
<p>Just in the last example, the introduction of the ART VM resolved this problem. Passing <code class="highlighter-rouge">--min-api 21</code> produces the more efficient sequence which only does a single <code class="highlighter-rouge">cmp-long</code>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ java -jar d8.jar \
--lib $ANDROID_HOME/platforms/android-28/android.jar \
--release \
--min-api 21 \
--output . \
*.class
$ $ANDROID_HOME/build-tools/28.0.3/dexdump -d classes.dex
[000138] LongCompare.somethingWithBytes:(J)V
0000: const-wide/16 v0, #int 0
0002: cmp-long v2, v2, v0
0004: if-ltz v2, 0009
0006: if-nez v2, 0008
…
</code></pre></div></div>
<p>Once again D8 changes the shape of the bytecodes it uses for the purpose of compatibility. When your application no longer supports the versions of Android which have the broken vendor implementations, the bytecode is updated to the more efficient form.</p>
<p>But while ART has brought a normalization to the VM across the ecosystem eliminating (or at least reducing) these vendor-specific bugs, it is not exempt from bugs itself.</p>
<h3 id="recursion">Recursion</h3>
<p>Bugs that occur in ART itself affect specific versions of Android regardless of the vendor. As D8 evolves and changes the bytecode it emits, dormant bugs in ART can suddenly surface.</p>
<p>The example which demonstrates an interesting bug is admittedly very contrived, but the code was derived from a real application and distilled into a self-contained example.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">java.util.List</span><span class="o">;</span>
<span class="kd">class</span> <span class="nc">Recursion</span> <span class="o">{</span>
<span class="kd">private</span> <span class="kt">void</span> <span class="nf">f</span><span class="o">(</span><span class="kt">int</span> <span class="n">x</span><span class="o">,</span> <span class="kt">double</span> <span class="n">y</span><span class="o">,</span> <span class="kt">double</span> <span class="n">u</span><span class="o">,</span> <span class="kt">double</span> <span class="n">v</span><span class="o">,</span> <span class="nc">List</span><span class="o"><</span><span class="nc">String</span><span class="o">></span> <span class="n">w</span><span class="o">)</span> <span class="o">{</span>
<span class="n">f</span><span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">y</span><span class="o">,</span> <span class="n">u</span><span class="o">,</span> <span class="n">v</span><span class="o">,</span> <span class="n">w</span><span class="o">);</span>
<span class="n">f</span><span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">y</span><span class="o">,</span> <span class="n">u</span><span class="o">,</span> <span class="n">v</span><span class="o">,</span> <span class="n">w</span><span class="o">);</span>
<span class="n">f</span><span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">y</span><span class="o">,</span> <span class="n">u</span><span class="o">,</span> <span class="n">v</span><span class="o">,</span> <span class="n">w</span><span class="o">);</span>
<span class="n">f</span><span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">y</span><span class="o">,</span> <span class="n">u</span><span class="o">,</span> <span class="n">v</span><span class="o">,</span> <span class="n">w</span><span class="o">);</span>
<span class="n">f</span><span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">y</span><span class="o">,</span> <span class="n">u</span><span class="o">,</span> <span class="n">v</span><span class="o">,</span> <span class="n">w</span><span class="o">);</span>
<span class="n">f</span><span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">y</span><span class="o">,</span> <span class="n">u</span><span class="o">,</span> <span class="n">v</span><span class="o">,</span> <span class="n">w</span><span class="o">);</span>
<span class="n">f</span><span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">y</span><span class="o">,</span> <span class="n">u</span><span class="o">,</span> <span class="n">v</span><span class="o">,</span> <span class="n">w</span><span class="o">);</span>
<span class="n">f</span><span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">y</span><span class="o">,</span> <span class="n">u</span><span class="o">,</span> <span class="n">v</span><span class="o">,</span> <span class="n">w</span><span class="o">);</span>
<span class="n">f</span><span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">y</span><span class="o">,</span> <span class="n">u</span><span class="o">,</span> <span class="n">v</span><span class="o">,</span> <span class="n">w</span><span class="o">);</span>
<span class="n">w</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">g</span><span class="o">(</span><span class="n">y</span><span class="o">,</span> <span class="n">u</span><span class="o">,</span> <span class="n">v</span><span class="o">));</span>
<span class="o">}</span>
<span class="kd">private</span> <span class="nc">String</span> <span class="nf">g</span><span class="o">(</span><span class="kt">double</span> <span class="n">y</span><span class="o">,</span> <span class="kt">double</span> <span class="n">u</span><span class="o">,</span> <span class="kt">double</span> <span class="n">v</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="kc">null</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>In Android 6.0 (API 23), ART’s ahead-of-time (AOT) compiler added call analysis in order to perform method inlining. Due to the heavily-recursive nature of the <code class="highlighter-rouge">f</code> method above, the <code class="highlighter-rouge">dex2oat</code> compiler will actually consume all of the memory on the device during this analysis and crash. This was fixed in the next release, Android 7.0 (API 24).</p>
<p>When your minimum SDK is below 24, D8 will change the dex file to work around this bug. But before looking at the workaround, let’s reproduce the crash.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ javac *.java
$ java -jar d8.jar \
--lib $ANDROID_HOME/platforms/android-28/android.jar \
--release \
--min-api 24 \
--output . \
*.class
</code></pre></div></div>
<p>We pass <code class="highlighter-rouge">--min-api 24</code> to D8 in order to produce a dex file that does not contain the workaround for the bug. If you push this dex file to an API 23 device, <code class="highlighter-rouge">dex2oat</code> will refuse to compile it.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ adb shell push classes.dex /sdcard
$ adb shell dex2oat --dex-file=/sdcard/classes.dex --oat-file=/sdcard/classes.oat
$ adb logcat
…
11-29 13:57:08.303 4508 4508 I dex2oat : dex2oat --dex-file=/sdcard/classes.dex --oat-file=/sdcard/classes.oat
11-29 13:57:08.306 4508 4508 W dex2oat : Failed to open .dex from file '/sdcard/classes.dex': Failed to open dex file '/sdcard/classes.dex' from memory: Unrecognized version number in /sdcard/classes.dex: 0 3 7
11-29 13:57:08.306 4508 4508 E dex2oat : Failed to open some dex files: 1
11-29 13:57:08.309 4508 4508 I dex2oat : dex2oat took 7.440ms (threads: 4)
</code></pre></div></div>
<p>The <a href="https://source.android.com/devices/tech/dalvik/dex-format#dex-file-magic">documentation for the dex file format</a> defines that the first 8 bytes should be the characters <code class="highlighter-rouge">DEX</code>, a newline character, three number characters indicating the version, and then a null byte. Because <code class="highlighter-rouge">--min-api 24</code> was specified, the dex file declares version <code class="highlighter-rouge">037</code>. Dumping the first few bytes of the dex file confirm this.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ xxd classes.dex | head -1
00000000: 6465 780a 3033 3700 e595 2d8c 49b5 d6b6 dex.037...-.I...
</code></pre></div></div>
<p>In order to get this dex file to install on an older device the version must be changed to <code class="highlighter-rouge">035</code>. Any hex editor can be used to do this. I used <code class="highlighter-rouge">xxd</code> again to convert from binary to hexadecimal, edited the hexidecimal in an editor (which I know how to exit), and then used <code class="highlighter-rouge">xxd</code> again to convert hexadecimal back to binary.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ xxd -p classes.dex > classes.hex
$ nano classes.hex # Change 303337 to 303335
$ xxd -p -r classes.hex > classes.dex
</code></pre></div></div>
<p>With the version changed this dex file will now compile on Android 6.0 devices but with a different result.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ adb shell push classes.dex /sdcard
$ adb shell dex2oat --dex-file=/sdcard/classes.dex --oat-file=/sdcard/classes.oat
Segmentation fault
</code></pre></div></div>
<p>Whoops! We (successfully) crashed the AOT compiler. Running <code class="highlighter-rouge">dex2oat</code> with the same dex file on Android 7.0 or newer does not trigger the crash, as expected.</p>
<p>Removing the <code class="highlighter-rouge">--min-api 24</code> line will force D8 to insert its work around for this AOT compiler problem. Before doing so the old dex file is renamed so that we can compare the two.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ mv classes.dex classes_api24.dex
$ java -jar d8.jar \
--lib $ANDROID_HOME/platforms/android-28/android.jar \
--release \
--output . \
*.class
</code></pre></div></div>
<p>Dumping the bytecodes of both shows the difference.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ $ANDROID_HOME/build-tools/28.0.3/dexdump -d classes_api24.dex
[000190] Recursion.f:(IDDDLjava/util/List;)V
0000: invoke-direct/range {v7, v8, v9, v10, v11, v12, v13, v14, v15}, LRecursion;.f:(IDDDLjava/util/List;)V
…
0018: invoke-direct/range {v7, v8, v9, v10, v11, v12, v13, v14, v15}, LRecursion;.f:(IDDDLjava/util/List;)V
001b: move-object v0, v7
001c: move-wide v1, v9
001d: move-wide v3, v11
001e: move-wide v5, v13
001f: invoke-direct/range {v0, v1, v2, v3, v4, v5, v6}, LRecursion;.g:(DDD)Ljava/lang/String;
0022: move-result-object v8
0023: invoke-interface {v15, v8}, Ljava/util/List;.add:(Ljava/lang/Object;)Z
0026: return-void
catches : (none)
$ $ANDROID_HOME/build-tools/28.0.3/dexdump -d classes.dex
[000198] Recursion.f:(IDDDLjava/util/List;)V
0000: invoke-direct/range {v7, v8, v9, v10, v11, v12, v13, v14, v15}, LRecursion;.f:(IDDDLjava/util/List;)V
…
0018: invoke-direct/range {v7, v8, v9, v10, v11, v12, v13, v14, v15}, LRecursion;.f:(IDDDLjava/util/List;)V
001b: move-object v0, v7
001c: move-wide v1, v9
001d: move-wide v3, v11
001e: move-wide v5, v13
001f: invoke-direct/range {v0, v1, v2, v3, v4, v5, v6}, LRecursion;.g:(DDD)Ljava/lang/String;
0022: move-result-object v8
0023: invoke-interface {v15, v8}, Ljava/util/List;.add:(Ljava/lang/Object;)Z
0026: return-void
0027: move-exception v8
0028: throw v8
catches : 1
0x0018 - 0x001b
Ljava/lang/Throwable; -> 0x0027
</code></pre></div></div>
<p>The contents of each version of the method are the exact same until the very end. The version which works around the bug has two extra bytecodes, <code class="highlighter-rouge">move-exception</code> and <code class="highlighter-rouge">throw</code>, and an entry in the <code class="highlighter-rouge">catches</code> section. This is the bytecode equivalent of a try-catch block that simply re-throws the exception. By inserting this try-catch block, the AOT compiler’s call analysis for method inling is disabled.</p>
<p>The range of the catch block only covers the last recursive call from bytecode index <code class="highlighter-rouge">0018</code> to <code class="highlighter-rouge">001b</code>. If you were remove a single call to <code class="highlighter-rouge">f</code> in the original source code, the level of recursion won’t be large enough to trigger the bug in the AOT compiler. Therefore the try-catch workaround only surrounds the recursive calls when they’re problematic.</p>
<p>The same code when dexed with the old <code class="highlighter-rouge">dx</code> compiler will not cause a crash on Android 6.0. This is because the bytecode is less efficient and uses more registers which prevents the inlining analysis from even running.</p>
<hr />
<p>The three examples above are a few cases of vendor- and version-specific bugs in Android’s VMs. Just like the language feature desugaring covered in the previous posts, D8 will only apply workarounds for these bugs when necessary based on your minimum API level.</p>
<p>The conditionals which control whether these are applied are at the bottom of a <a href="https://r8.googlesource.com/r8/+/master/src/main/java/com/android/tools/r8/utils/InternalOptions.java">file named <code class="highlighter-rouge">InternalOptions.java</code></a> in the D8 codebase. Bugs in the VM aren’t only found in old versions of Android. If you search for <code class="highlighter-rouge">AndroidApiLevel.Q</code> in that file you’ll find two workarounds for VM bugs present in every version of Android (at least at time of writing).</p>
<p>It’s important to remember that all of these problems weren’t caused by D8. They were uncovered by D8 in its effort to use registers more effectively and order bytecodes more efficiently when compared to <code class="highlighter-rouge">dx</code>. For optimizing dex even further, we have to turn to D8’s optimizing sibling, R8, which we’ll start to examine in the next post.</p>
<p><em>(This post was adapted from a part of my <a href="/digging-into-d8-and-r8">Digging into D8 and R8</a> talk that was only partially presented. Watch the video and look out for future blog posts for more content like this.)</em></p>
Android's Java 9, 10, 11, and 12 Support2018-11-27T00:00:00+00:00https://jakewharton.com/androids-java-9-10-11-and-12-support<blockquote>
<p>Note: This post is part of a series on D8 and R8, Android’s new dexer and optimizer, respectively. For an intro to D8 read <a href="/androids-java-8-support/">“Android’s Java 8 support”</a>.</p>
</blockquote>
<p>The first post in this series <a href="/androids-java-8-support">explored Android’s Java 8 support</a>. Having support for the language features and APIs of Java 8 is table stakes at this point. We’re not quite there with the APIs yet, sadly, but D8 has us covered with the language features. There’s a future promise for the APIs which is essential for the health of the ecosystem.</p>
<p>A lot of the reaction to the previous post echoed that Java 8 is quite old. The rest of the Java ecosystem is starting to move to Java 11 (being the first long-term supported release after 8) after having toyed with Java 9 and 10. I was <em>hoping</em> for that reaction because I mostly wrote that post so that I could set up this one.</p>
<p>With Java releases happening more frequently, Android’s yearly release schedule and delayed uptake of newer language features and APIs <em>feels</em> more painful. But is it actually the case that we’re stuck with those of Java 8? Let’s take a look at the Java releases beyond 8 and see how the Android toolchain fares.</p>
<h3 id="java-9">Java 9</h3>
<p>The last release on the 2 - 3 year schedule, Java 9 contains a few new language features. None of them are major like lambdas were. Instead, this release focused on cleaning up some of the sharp edges on existing features.</p>
<h4 id="concise-try-with-resources">Concise Try With Resources</h4>
<p>Prior to this release the try-with-resources construct required that you define a local variable (such as <code class="highlighter-rouge">try (Closeable bar = foo.bar())</code>). But if you already have a <code class="highlighter-rouge">Closeable</code>, defining a new variable is redundant. As such, this release allows you to omit declaring a new variable if you already have an effectively-final reference.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">java.io.*</span><span class="o">;</span>
<span class="kd">class</span> <span class="nc">Java9TryWithResources</span> <span class="o">{</span>
<span class="nc">String</span> <span class="nf">effectivelyFinalTry</span><span class="o">(</span><span class="nc">BufferedReader</span> <span class="n">r</span><span class="o">)</span> <span class="kd">throws</span> <span class="nc">IOException</span> <span class="o">{</span>
<span class="k">try</span> <span class="o">(</span><span class="n">r</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">r</span><span class="o">.</span><span class="na">readLine</span><span class="o">();</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>This feature is implemented entirely in the Java compiler so D8 is able to dex it for Android.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ javac *.java
$ java -jar d8.jar \
--lib $ANDROID_HOME/platforms/android-28/android.jar \
--release \
--output . \
*.class
$ ls
Java9TryWithResources.java Java9TryWithResources.class classes.dex
</code></pre></div></div>
<p>Unlike the lambdas or static interface methods of Java 8 which required special desugaring, this Java 9 feature becomes available to all API levels for free.</p>
<h4 id="anonymous-diamond">Anonymous Diamond</h4>
<p>Java 7 introduced the diamond operator which allowed omitting a generic type from the initializer if it could be inferred from the variable type.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">List</span><span class="o"><</span><span class="nc">String</span><span class="o">></span> <span class="n">strings</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ArrayList</span><span class="o"><>();</span>
</code></pre></div></div>
<p>This cut down on redundant declarations, but it wasn’t available for use on anonymous classes. With Java 9 that is now supported.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">java.util.concurrent.*</span><span class="o">;</span>
<span class="kd">class</span> <span class="nc">Java9AnonymousDiamond</span> <span class="o">{</span>
<span class="nc">Callable</span><span class="o"><</span><span class="nc">String</span><span class="o">></span> <span class="nf">anonymousDiamond</span><span class="o">()</span> <span class="o">{</span>
<span class="nc">Callable</span><span class="o"><</span><span class="nc">String</span><span class="o">></span> <span class="n">call</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">Callable</span><span class="o"><>()</span> <span class="o">{</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="nc">String</span> <span class="nf">call</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="s">"Hey"</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">};</span>
<span class="k">return</span> <span class="n">call</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Once again this is entirely implemented in the Java compiler so the resulting bytecode is as if <code class="highlighter-rouge">String</code> was explicitly specified.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ javac *.java
$ javap -c *.class
class Java9AnonymousDiamond {
java.util.concurrent.Callable<java.lang.String> anonymousDiamond();
Code:
0: new #7 // class Java9AnonymousDiamond$1
3: dup
4: aload_0
5: invokespecial #8 // Method Java9AnonymousDiamond$1."<init>":(LJava9AnonymousDiamond;)V
8: areturn
}
class Java9AnonymousDiamond$1 implements java.util.concurrent.Callable<java.lang.String> {
final Java9AnonymousDiamond this$0;
Java9AnonymousDiamond$1(Java9AnonymousDiamond);
Code:
0: aload_0
1: aload_1
2: putfield #1 // Field this$0:LJava9AnonymousDiamond;
5: aload_0
6: invokespecial #2 // Method java/lang/Object."<init>":()V
9: return
public java.lang.String call();
Code:
0: ldc #3 // String Hey
2: areturn
}
</code></pre></div></div>
<p>Because there is nothing interesting in the bytecode, D8 handles this without issue.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ java -jar d8.jar \
--lib $ANDROID_HOME/platforms/android-28/android.jar \
--release \
--output . \
*.class
$ ls
Java9AnonymousDiamond.java Java9AnonymousDiamond.class Java9AnonymousDiamond$1.class classes.dex
</code></pre></div></div>
<p>Yet another language feature available to all API levels for free.</p>
<h4 id="private-interface-methods">Private Interface Methods</h4>
<p>Interfaces with multiple static or default methods can often lead to duplicated code in their bodies. If these methods were part of a class and not an interface private helper functions could be extracted. Java 9 adds the ability for interfaces to contain private methods which are only accessible to its static and default methods.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">interface</span> <span class="nc">Java9PrivateInterface</span> <span class="o">{</span>
<span class="kd">static</span> <span class="nc">String</span> <span class="nf">hey</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="nf">getHey</span><span class="o">();</span>
<span class="o">}</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="nc">String</span> <span class="nf">getHey</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="s">"hey"</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>This is the first language feature that requires some kind of support. Prior to this release, the <code class="highlighter-rouge">private</code> modifier was not allowed on an interface member. Since D8 is already responsible for desugaring default and static methods, private methods were straightforward to include using the same technique.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ javac *.java
$ java -jar d8.jar \
--lib $ANDROID_HOME/platforms/android-28/android.jar \
--release \
--output . \
*.class
$ ls
Java9PrivateInterface.java Java9PrivateInterface.class classes.dex
</code></pre></div></div>
<p>Static and default methods are supported natively in ART as of API 24. When you pass <code class="highlighter-rouge">--min-api 24</code> for this example, the static method is not desugared. Curiously, though, the private static method is also not desugared.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ $ANDROID_HOME/build-tools/28.0.2/dexdump -d classes.dex
Class #1 -
Class descriptor : 'LJava9PrivateInterface;'
Access flags : 0x0600 (INTERFACE ABSTRACT)
Superclass : 'Ljava/lang/Object;'
Direct methods -
#0 : (in LJava9PrivateInterface;)
name : 'getHey'
type : '()Ljava/lang/String;'
access : 0x000a (PRIVATE STATIC)
00047c: |[00047c] Java9PrivateInterface.getHey:()Ljava/lang/String;
00048c: 1a00 2c00 |0000: const-string v0, "hey"
000490: 1100 |0002: return-object v0
</code></pre></div></div>
<p>We can see that the <code class="highlighter-rouge">getHey()</code> method’s access flags still contain both <code class="highlighter-rouge">PRIVATE</code> and <code class="highlighter-rouge">STATIC</code>. If you add a <code class="highlighter-rouge">main</code> method which calls <code class="highlighter-rouge">hey()</code> and push this to a device it will actually work. Despite being a feature added in Java 9, ART allows private interface members since API 24!</p>
<p>Those are all the language features of Java 9 and they all already work on Android. How about that.</p>
<p>The APIs of Java 9, though, are not yet included in the Android SDK. A new process API, var handles, a version of the Reactive Streams interfaces, and collection factories are just some of those which were added. Since libcore (which contains implementation of <code class="highlighter-rouge">java.*</code>) and ART are developed in AOSP, we can peek and see that work is already underway towards supporting Java 9. Once included included in the SDK, some of its APIs will be candidates for desugaring to all API levels.</p>
<h4 id="string-concat">String Concat</h4>
<p>The new language features and APIs of a Java release tend to be what we talk about most. But each release is also an opportunity to optimize the bytecode which is used to implement a feature. Java 9 brought an optimization to a ubiquitous language feature: string concatenation.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Java9Concat</span> <span class="o">{</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="nc">String</span> <span class="nf">thing</span><span class="o">(</span><span class="nc">String</span> <span class="n">a</span><span class="o">,</span> <span class="nc">String</span> <span class="n">b</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="s">"A: "</span> <span class="o">+</span> <span class="n">a</span> <span class="o">+</span> <span class="s">" and B: "</span> <span class="o">+</span> <span class="n">b</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>If we take this fairly innocuous piece of code and compile it with Java 8 the resulting bytecode will use a <code class="highlighter-rouge">StringBuilder</code>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ java -version
java version "1.8.0_192"
Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
$ javac *.java
$ javap -c *.class
class Java9Concat {
public static java.lang.String thing(java.lang.String, java.lang.String);
Code:
0: new #2 // class java/lang/StringBuilder
3: dup
4: invokespecial #3 // Method java/lang/StringBuilder."<init>":()V
7: ldc #4 // String A:
9: invokevirtual #5 // Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
12: aload_0
13: invokevirtual #5 // Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
16: ldc #6 // String and B:
18: invokevirtual #5 // Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
21: aload_1
22: invokevirtual #5 // Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
25: invokevirtual #7 // Method java/lang/StringBuilder.toString:()Ljava/lang/String;
28: areturn
}
</code></pre></div></div>
<p>The bytecode contains the code we otherwise would have written if the language didn’t allow simple concatenation.</p>
<p>If we change the compiler to Java 9, however, the result is very different.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ java -version
java version "9.0.1"
Java(TM) SE Runtime Environment (build 9.0.1+11)
Java HotSpot(TM) 64-Bit Server VM (build 9.0.1+11, mixed mode)
$ javac *.java
$ javap -c *.class
class Java9Concat {
public static java.lang.String thing(java.lang.String, java.lang.String);
Code:
0: aload_0
1: aload_1
2: invokedynamic #2, 0 // InvokeDynamic #0:makeConcatWithConstants:(
Ljava/lang/String;Ljava/lang/String;)Ljava/lang/String;
7: areturn
}
</code></pre></div></div>
<p>The entire <code class="highlighter-rouge">StringBuilder</code> usage has been replaced with a single <code class="highlighter-rouge">invokedynamic</code> bytecode! The behavior here is similar to how <a href="/androids-java-8-support/#native-lambdas">native lambdas work on the JVM</a> which was discussed in the last post.</p>
<p>At runtime, on the JVM, the <a href="https://docs.oracle.com/javase/9/docs/api/java/lang/invoke/StringConcatFactory.html#makeConcatWithConstants-java.lang.invoke.MethodHandles.Lookup-java.lang.String-java.lang.invoke.MethodType-java.lang.String-java.lang.Object...-">JDK class <code class="highlighter-rouge">StringConcatFactory</code></a> is responsible for returning a block of code which can efficiently concatenate the arguments and constants together. This allows the implementation to change over time without the code having to be recompiled. It also means that the <code class="highlighter-rouge">StringBuilder</code> can be pre-sized more accurately since the argument’s lengths can be queried.</p>
<p>If you want to learn more about why this change was made, <a href="https://www.youtube.com/watch?v=wIyeOaitmWM">Aleksey Shipilëv gave a great presentation</a> on the motivations, implementation, and resulting benchmarks of the change.</p>
<p>Since the Android APIs don’t yet include anything from Java 9, there is no <code class="highlighter-rouge">StringConcatFactory</code> available at runtime. Thankfully, just like it did for <code class="highlighter-rouge">LambdaMetafactory</code> and lambdas, D8 is able to desugar <code class="highlighter-rouge">StringConcatFactory</code> for concatenations.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ java -jar d8.jar \
--lib $ANDROID_HOME/platforms/android-28/android.jar \
--release \
--output . \
*.class
$ $ANDROID_HOME/build-tools/28.0.2/dexdump -d classes.dex
[000144] Java9Concat.thing:(Ljava/lang/String;Ljava/lang/String;)Ljava/lang/String;
0000: new-instance v0, Ljava/lang/StringBuilder;
0002: invoke-direct {v0}, Ljava/lang/StringBuilder;.<init>:()V
0005: const-string v1, "A: "
0007: invoke-virtual {v0, v1}, Ljava/lang/StringBuilder;.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
000a: invoke-virtual {v0, v2}, Ljava/lang/StringBuilder;.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
000d: const-string v2, " and B: "
000f: invoke-virtual {v0, v2}, Ljava/lang/StringBuilder;.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
0012: invoke-virtual {v0, v3}, Ljava/lang/StringBuilder;.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
0015: invoke-virtual {v0}, Ljava/lang/StringBuilder;.toString:()Ljava/lang/String;
0018: move-result-object v2
0019: return-object v2
</code></pre></div></div>
<p>This means that all of the language features of Java 9 can be used on all API levels of Android despite changes in the bytecode that the Java compiler emits.</p>
<p>But Java is now on a six-month release schedule making Java 9 actually two versions old. Can we keep it going with newer versions?</p>
<h3 id="java-10">Java 10</h3>
<p>The only language feature of Java 10 was called local-variable type inference. This allows you to omit the type of local variable by replacing it with <code class="highlighter-rouge">var</code> when that type can be inferred.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">java.util.*</span><span class="o">;</span>
<span class="kd">class</span> <span class="nc">Java10</span> <span class="o">{</span>
<span class="nc">List</span><span class="o"><</span><span class="nc">String</span><span class="o">></span> <span class="nf">localVariableTypeInferrence</span><span class="o">()</span> <span class="o">{</span>
<span class="kt">var</span> <span class="n">url</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ArrayList</span><span class="o"><</span><span class="nc">String</span><span class="o">>();</span>
<span class="k">return</span> <span class="n">url</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>This is another feature implemented entirely in the Java compiler.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ javac *.java
$ javap -c *.class
Compiled from "Java10.java"
class Java10 {
java.util.List<java.lang.String> localVariableTypeInferrence();
Code:
0: new #2 // class java/util/ArrayList
3: dup
4: invokespecial #3 // Method java/util/ArrayList."<init>":()V
7: areturn
}
</code></pre></div></div>
<p>No new bytecodes or runtime APIs are required for this feature to work and so it can be used for Android just fine.</p>
<p>Of course, like the versions of Java before it, there are new APIs in this release such as <code class="highlighter-rouge">Optional.orElseThrow</code>, <code class="highlighter-rouge">List.copyOf</code>, and <code class="highlighter-rouge">Collectors.toUnmodifiableList</code>. Once added to the Android SDK in a future API level, these APIs can be trivially desugared to run on all API levels.</p>
<h3 id="java-11">Java 11</h3>
<p>Local-variable type inference was enhanced in Java 11 to support its use on lambda variables. You don’t see types used in lambda parameters often so a lot of people don’t even know this syntax exists. This is useful when you need to provide an explicit type to help type inference or when you want to use a type-annotation on the parameter.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">java.util.function.*</span><span class="o">;</span>
<span class="nd">@interface</span> <span class="nc">NonNull</span> <span class="o">{}</span>
<span class="kd">class</span> <span class="nc">Java11</span> <span class="o">{</span>
<span class="kt">void</span> <span class="nf">lambdaParameterTypeInferrence</span><span class="o">()</span> <span class="o">{</span>
<span class="nc">Function</span><span class="o"><</span><span class="nc">String</span><span class="o">,</span> <span class="nc">String</span><span class="o">></span> <span class="n">func</span> <span class="o">=</span> <span class="o">(</span><span class="nd">@NonNull</span> <span class="kt">var</span> <span class="n">x</span><span class="o">)</span> <span class="o">-></span> <span class="n">x</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Just like Java 10’s local-variable type inference this feature is implemented entirely in the Java compiler allowing it to work on Android.</p>
<p>New APIs in Java 11 include a bunch of new helpers on <code class="highlighter-rouge">String</code>, <code class="highlighter-rouge">Predicate.not</code>, and null factories for <code class="highlighter-rouge">Reader</code>, <code class="highlighter-rouge">Writer</code>, <code class="highlighter-rouge">InputSteam</code>, and <code class="highlighter-rouge">OutputStream</code>. Nearly all of the API additions in this release could be trivially desugared once available.</p>
<p>A major API addition to Java 11 is the <a href="https://docs.oracle.com/en/java/javase/11/docs/api/java.net.http/java/net/http/package-summary.html">new HTTP client, <code class="highlighter-rouge">java.net.http</code></a>. This client was previously available experimentally in the <code class="highlighter-rouge">jdk.incubator.http</code> package since Java 9. This is a very large API surface and implementation which leverages <code class="highlighter-rouge">CompletableFuture</code> extensively. It will be interesting to see whether or not this even lands in the Android SDK let alone is available via desugaring.</p>
<h4 id="nestmates">Nestmates</h4>
<p>Like Java 9 and its string concatenation bytecode optimization, Java 11 took the opportunity to fix a long-standing disparity between Java’s source code and its class files and the JVM: nested classes.</p>
<p>In Java 1.1, nested classes were added to the language but not the class specification or JVM. In order to work around the lack of support in class file, nesting classes in a source file instead creates sibling classes which use a naming convention to convey nesting.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Outer</span> <span class="o">{</span>
<span class="kd">class</span> <span class="nc">Inner</span> <span class="o">{}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Compiling this with Java 10 or earlier will produce two class files from a single source file.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ java -version
java version "10" 2018-03-20
Java(TM) SE Runtime Environment 18.3 (build 10+46)
Java HotSpot(TM) 64-Bit Server VM 18.3 (build 10+46, mixed mode)
$ javac *.java
$ ls
Outer.java Outer.class Outer$Inner.class
</code></pre></div></div>
<p>As far as the JVM is concerned, these classes have no relationship except that they exist in the same package.</p>
<p>This illusion mostly works. Where it starts to break down is when one of the classes needs to access something that is private in the other.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Outer</span> <span class="o">{</span>
<span class="kd">private</span> <span class="nc">String</span> <span class="n">name</span><span class="o">;</span>
<span class="kd">class</span> <span class="nc">Inner</span> <span class="o">{</span>
<span class="nc">String</span> <span class="nf">sayHi</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="s">"Hi, "</span> <span class="o">+</span> <span class="n">name</span> <span class="o">+</span> <span class="s">"!"</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>When these classes are made siblings, <code class="highlighter-rouge">Outer$Inner.sayHi()</code> is unable to access <code class="highlighter-rouge">Outer.name</code> because it is private to another class.</p>
<p>In order to work around this problem and maintain the nesting illusion, the Java compiler adds a package-private <em>synthetic accessor method</em> for any member accessed across this boundary.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> class Outer {
private String name;
<span class="gi">+
+ String access$000() {
+ return name;
+ }
</span>
class Inner {
String sayHi() {
<span class="gd">- return "Hi, " + name + "!";
</span><span class="gi">+ return "Hi, " + access$000() + "!";
</span> }
</code></pre></div></div>
<p>This is visible in the compiled class file for <code class="highlighter-rouge">Outer</code>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ javap -c -p Outer.class
class Outer {
private java.lang.String name;
static java.lang.String access$000(Outer);
Code:
0: aload_0
1: getfield #1 // Field name:Ljava/lang/String;
4: areturn
}
</code></pre></div></div>
<p>Historically this has been at most a small annoyance on the JVM. For Android, though, these synthetic accessor methods contribute to the method count in our dex files, increase APK size, slow down class loading and verification, and degrade performance by turning a field lookup into a method call!</p>
<p>In Java 11, the class file format was updated to introduce the concept of <em>nests</em> to describe these nesting relationships.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ java -version
java version "11.0.1" 2018-10-16 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.1+13-LTS)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.1+13-LTS, mixed mode)
$ javac *.java
$ javap -v -p *.class
class Outer {
private java.lang.String name;
}
NestMembers:
Outer$Inner
class Outer$Inner {
final Outer this$0;
Outer$Inner(Outer);
Code: …
java.lang.String sayHi();
Code: …
}
NestHost: class Outer
</code></pre></div></div>
<p>The output here has been trimmed significantly, but the two class files are still produced except without an <code class="highlighter-rouge">access$000</code> in <code class="highlighter-rouge">Outer</code> and with new <code class="highlighter-rouge">NestMembers</code> and <code class="highlighter-rouge">NestHost</code> attributes. These allow the VM to enforce a level of access control between package-private and private called <em>nestmates</em>. As a result, <code class="highlighter-rouge">Inner</code> can directly access <code class="highlighter-rouge">Outer</code>’s <code class="highlighter-rouge">name</code> field.</p>
<p>ART does not understand the concept of nestmates so it needs to be desugared back into synthetic accessor methods.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ java -jar d8.jar \
--lib $ANDROID_HOME/platforms/android-28/android.jar \
--release \
--output . \
*.class
Compilation failed with an internal error.
java.lang.UnsupportedOperationException
at com.android.tools.r8.org.objectweb.asm.ClassVisitor.visitNestHostExperimental(ClassVisitor.java:158)
at com.android.tools.r8.org.objectweb.asm.ClassReader.accept(ClassReader.java:541)
at com.android.tools.r8.org.objectweb.asm.ClassReader.accept(ClassReader.java:391)
at com.android.tools.r8.graph.JarClassFileReader.read(JarClassFileReader.java:107)
at com.android.tools.r8.dex.ApplicationReader$ClassReader.lambda$readClassSources$1(ApplicationReader.java:231)
at java.base/java.util.concurrent.ForkJoinTask$AdaptedCallable.exec(ForkJoinTask.java:1448)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
</code></pre></div></div>
<p>Unfortunately, at the time of writing, this does not work. The version of ASM, the library used to read Java class files, predates the final implementation of nestmates. Beyond that, though, D8 does not support desugaring of nest mates. You can <a href="https://issuetracker.google.com/issues/116628246">star the D8 feature request</a> on the Android issue tracker to convey your support for this feature.</p>
<p>Without support for desugaring nestmates it is currently impossible to use Java 11 for Android. Even if you avoid accessing things across the nested boundary, the mere presence of nesting will fail to compile.</p>
<p>Without the APIs from Java 11 in the Android SDK, its single language feature of lambda parameter type inference isn’t compelling. For now, Android developers are not missing anything by being stuck on Java 10. That is, until we start looking forward…</p>
<h3 id="java-12">Java 12</h3>
<p>With a release date of March 2019, Java 12 is quickly approaching. The language features and APIs of this release have been in development for a few months already. Through early-access builds, we can download and experiment with these today.</p>
<p>In the current EA build, number 20, there are two new language features available: expression switch and string literals.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Java12</span> <span class="o">{</span>
<span class="kd">static</span> <span class="kt">int</span> <span class="nf">letterCount</span><span class="o">(</span><span class="nc">String</span> <span class="n">s</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="nf">switch</span> <span class="o">(</span><span class="n">s</span><span class="o">)</span> <span class="o">{</span>
<span class="k">case</span> <span class="s">"one"</span><span class="o">,</span> <span class="s">"two"</span> <span class="o">-></span> <span class="mi">3</span><span class="o">;</span>
<span class="k">case</span> <span class="s">"three"</span> <span class="o">-></span> <span class="mi">5</span><span class="o">;</span>
<span class="k">default</span> <span class="o">-></span> <span class="n">s</span><span class="o">.</span><span class="na">length</span><span class="o">();</span>
<span class="o">};</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">...</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="err">`</span>
<span class="n">__</span> <span class="n">______</span> <span class="n">______</span> <span class="n">______</span> <span class="n">______</span> <span class="n">______</span> <span class="n">______</span>
<span class="o">/</span><span class="err">\</span> <span class="err">\</span> <span class="o">/</span><span class="err">\</span> <span class="n">___</span><span class="err">\</span> <span class="o">/</span><span class="err">\</span><span class="n">__</span> <span class="n">_</span><span class="err">\</span> <span class="o">/</span><span class="err">\</span><span class="n">__</span> <span class="n">_</span><span class="err">\</span> <span class="o">/</span><span class="err">\</span> <span class="n">___</span><span class="err">\</span> <span class="o">/</span><span class="err">\</span> <span class="o">==</span> <span class="err">\</span> <span class="o">/</span><span class="err">\</span> <span class="n">___</span><span class="err">\</span>
<span class="err">\</span> <span class="err">\</span> <span class="err">\</span><span class="n">____</span> <span class="err">\</span> <span class="err">\</span> <span class="n">__</span><span class="err">\</span> <span class="err">\</span><span class="o">/</span><span class="n">_</span><span class="o">/</span><span class="err">\</span> <span class="err">\</span><span class="o">/</span> <span class="err">\</span><span class="o">/</span><span class="n">_</span><span class="o">/</span><span class="err">\</span> <span class="err">\</span><span class="o">/</span> <span class="err">\</span> <span class="err">\</span> <span class="n">__</span><span class="err">\</span> <span class="err">\</span> <span class="err">\</span> <span class="n">__</span><span class="o"><</span> <span class="err">\</span> <span class="err">\</span><span class="n">___</span> <span class="err">\</span>
<span class="err">\</span> <span class="err">\</span><span class="n">_____</span><span class="err">\</span> <span class="err">\</span> <span class="err">\</span><span class="n">_____</span><span class="err">\</span> <span class="err">\</span> <span class="err">\</span><span class="n">_</span><span class="err">\</span> <span class="err">\</span> <span class="err">\</span><span class="n">_</span><span class="err">\</span> <span class="err">\</span> <span class="err">\</span><span class="n">_____</span><span class="err">\</span> <span class="err">\</span> <span class="err">\</span><span class="n">_</span><span class="err">\</span> <span class="err">\</span><span class="n">_</span><span class="err">\</span> <span class="err">\</span><span class="o">/</span><span class="err">\</span><span class="n">_____</span><span class="err">\</span>
<span class="err">\</span><span class="o">/</span><span class="n">_____</span><span class="o">/</span> <span class="err">\</span><span class="o">/</span><span class="n">_____</span><span class="o">/</span> <span class="err">\</span><span class="o">/</span><span class="n">_</span><span class="o">/</span> <span class="err">\</span><span class="o">/</span><span class="n">_</span><span class="o">/</span> <span class="err">\</span><span class="o">/</span><span class="n">_____</span><span class="o">/</span> <span class="err">\</span><span class="o">/</span><span class="n">_</span><span class="o">/</span> <span class="o">/</span><span class="n">_</span><span class="o">/</span> <span class="err">\</span><span class="o">/</span><span class="n">_____</span><span class="o">/</span>
<span class="err">`</span><span class="o">);</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"three: "</span> <span class="o">+</span> <span class="n">letterCount</span><span class="o">(</span><span class="s">"three"</span><span class="o">));</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Once again, both of these features are implemented entirely as part of the Java compiler without any new bytecodes or APIs.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ java -version
openjdk version "12-ea" 2019-03-19
OpenJDK Runtime Environment (build 12-ea+20)
OpenJDK 64-Bit Server VM (build 12-ea+20, mixed mode, sharing)
$ javac *.java
$ java -jar d8.jar \
--lib $ANDROID_HOME/platforms/android-28/android.jar \
--release \
--output . \
*.class
$ ls
Java12.java Java12.class classes.dex
</code></pre></div></div>
<p>We can push this to a device to ensure that it actually works at runtime.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ adb push classes.dex /sdcard
classes.dex: 1 file pushed. 0.6 MB/s (1792 bytes in 0.003s)
$ adb shell dalvikvm -cp /sdcard/classes.dex Java12
__ ______ ______ ______ ______ ______ ______
/\ \ /\ ___\ /\__ _\ /\__ _\ /\ ___\ /\ == \ /\ ___\
\ \ \____ \ \ __\ \/_/\ \/ \/_/\ \/ \ \ __\ \ \ __< \ \___ \
\ \_____\ \ \_____\ \ \_\ \ \_\ \ \_____\ \ \_\ \_\ \/\_____\
\/_____/ \/_____/ \/_/ \/_/ \/_____/ \/_/ /_/ \/_____/
three: 5
</code></pre></div></div>
<p>This works because the bytecode for expression switch is the same as the “regular” switch we would otherwise write with an uninitialized local, <code class="highlighter-rouge">case</code> blocks with <code class="highlighter-rouge">break</code>, and a separate <code class="highlighter-rouge">return</code> statement. And a multi-line string literal is just a string with newlines in it, something we’ve been able to do with escape characters forever.</p>
<p>As with all the other releases covered, there will be new APIs in Java 12 and it’s the same story as before. They’ll need added to the Android SDK and evaluated for desugaring capability.</p>
<p>Hopefully by the time Java 12 is actually released D8 will have implemented desugaring for Java 11’s nestmates. Otherwise the pain of being stuck on Java 10 will go up quite a bit!</p>
<hr />
<p>Java 8 language features are here and desugaring of its APIs are coming (<a href="https://issuetracker.google.com/issues/114481425">star the issue!</a>). As the larger Java ecosystem moves forward to newer versions, it’s reassuring that every language feature between 8 and 12 is already available on Android.</p>
<p>With Java 9 work seemingly happening in AOSP (cross your fingers for Android P+1), hopefully we’ll have a new batch of APIs in the summer as candidates for desugaring. Once that lands, the smaller releases of Java will hopefully yield faster integration into the Android SDK.</p>
<p>Despite this, the end advice remains the same as in the last post. It’s vitally important to maintain pressure on Android for supporting the new APIs and VM features from newer versions of Java. Without APIs being integrated into the SDK they can’t (easily) be made available for use via desugaring. Without VM features being integrated into ART D8 bears a desugaring burden for all API levels instead of only to provide backwards compatibility.</p>
<p>Before these posts move on to talk about R8, the optimizing version of D8, the next one will cover how D8 works around version-specific and vendor-specific bugs in the VM.</p>
<p><em>(This post was adapted from a part of my <a href="/digging-into-d8-and-r8">Digging into D8 and R8</a> talk that was never presented. Watch the video and look out for future blog posts for more content like this.)</em></p>
Android's Java 8 Support2018-11-20T00:00:00+00:00https://jakewharton.com/androids-java-8-support<p>I’ve worked from home for a few years, and during that time I’ve heard people around the office complaining about Android’s varying support for different versions of Java. Every year at Google I/O you could find me asking about it at the fireside chats or directly to the folks responsible. At conferences and other developer events it comes up in conversation or in talks with different degrees of accuracy. It’s a complicated topic because what exactly we mean when talking about Android’s Java support can be unclear. There’s a lot to a single version of Java: the language features, the bytecode, the tools, the APIs, the JVM, and more.</p>
<p>When someone talks about Android’s Java 8 support they usually are referring to the language features. So let’s start there with a look at how Android’s toolchain deals with the language features of Java 8.</p>
<h3 id="lambdas">Lambdas</h3>
<p>The banner language feature of Java 8 was by far the addition of lambdas. This brought a more terse expression of code as data whereas previously more verbose constructs like anonymous classes would be used.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Java8</span> <span class="o">{</span>
<span class="kd">interface</span> <span class="nc">Logger</span> <span class="o">{</span>
<span class="kt">void</span> <span class="nf">log</span><span class="o">(</span><span class="nc">String</span> <span class="n">s</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">...</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
<span class="n">sayHi</span><span class="o">(</span><span class="n">s</span> <span class="o">-></span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">s</span><span class="o">));</span>
<span class="o">}</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">sayHi</span><span class="o">(</span><span class="nc">Logger</span> <span class="n">logger</span><span class="o">)</span> <span class="o">{</span>
<span class="n">logger</span><span class="o">.</span><span class="na">log</span><span class="o">(</span><span class="s">"Hello!"</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>After compiling this program with <code class="highlighter-rouge">javac</code>, running it through the legacy <code class="highlighter-rouge">dx</code> tool produces an error.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ javac *.java
$ ls
Java8.java Java8.class Java8$Logger.class
$ $ANDROID_HOME/build-tools/28.0.2/dx --dex --output . *.class
Uncaught translation error: com.android.dx.cf.code.SimException:
ERROR in Java8.main:([Ljava/lang/String;)V:
invalid opcode ba - invokedynamic requires --min-sdk-version >= 26
(currently 13)
1 error; aborting
</code></pre></div></div>
<p>This is because lambdas use a newer bytecode, <code class="highlighter-rouge">invokedynamic</code>, added in Java 7. As the error message indicates, Android’s support for this bytecode requires a minimum API of 26 or newer–something practically unfathomable for applications at the time of writing. Instead, a process named <em>desugaring</em> is used which turns lambdas into representations compatible with all API levels developers are targeting.</p>
<h3 id="desugaring-history">Desugaring History</h3>
<p>This history of the Android toolchain’s desugaring capability is… colorful. The goal is always the same: allow newer language features to run on all devices.</p>
<p>Initially a third-party tool called <a href="https://github.com/luontola/retrolambda">Retrolambda</a> had to be used. This worked by using the built-in mechanism which the JVM uses to turn lambdas into classes at runtime except happening at compile-time. The generated classes were very expensive in terms of method count, but work on the tool over time reduced the cost to something reasonable.</p>
<p>The Android tools team then <a href="https://android-developers.googleblog.com/2014/12/hello-world-meet-our-new-experimental.html">announced a new compiler</a> which would provide Java 8 language feature desugaring along with better performance. This was built on the Eclipse Java compiler but emitting Dalvik bytecode instead of Java bytecode. The Java 8 desugaring was extremely efficient, but otherwise adoption was low, performance was worse, and integration with other tooling was non-existent.</p>
<p>When the new compiler was (thankfully) abandoned, a Java bytecode to Java bytecode transformer which performed desugaring was <a href="https://android-developers.googleblog.com/2017/04/java-8-language-features-support-update.html">integrated into the Android Gradle plugin</a> from Bazel, Google’s bespoke build system. The desugaring output remained efficient but performance still wasn’t great. It was eventually made incremental, but work was happening concurrently to provide a better solution.</p>
<p>The <a href="https://android-developers.googleblog.com/2017/08/next-generation-dex-compiler-now-in.html">D8 dexer was announced</a> to replace the legacy <code class="highlighter-rouge">dx</code> tool with a promise of having desugar occur during dexing rather than a standalone Java bytecode transformation. The performance and accuracy of D8 compared to <code class="highlighter-rouge">dx</code> was a big win and it brought with it more efficient desugared bytecode. It was made the default dexer in Android Gradle plugin 3.1 and it then became responsible for desugaring in 3.2.</p>
<h3 id="d8">D8</h3>
<p>Using D8 to compile the above example to Dalvik bytecode succeeds.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ java -jar d8.jar \
--lib $ANDROID_HOME/platforms/android-28/android.jar \
--release \
--output . \
*.class
$ ls
Java8.java Java8.class Java8$Logger.class classes.dex
</code></pre></div></div>
<p>To see how D8 desugared the lambda we can use the <code class="highlighter-rouge">dexdump</code> tool which is part of the Android SDK. The tool produces quite a lot of output so we’ll only look at the relevant sections.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ $ANDROID_HOME/build-tools/28.0.2/dexdump -d classes.dex
[0002d8] Java8.main:([Ljava/lang/String;)V
0000: sget-object v0, LJava8$1;.INSTANCE:LJava8$1;
0002: invoke-static {v0}, LJava8;.sayHi:(LJava8$Logger;)V
0005: return-void
[0002a8] Java8.sayHi:(LJava8$Logger;)V
0000: const-string v0, "Hello"
0002: invoke-interface {v1, v0}, LJava8$Logger;.log:(Ljava/lang/String;)V
0005: return-void
…
</code></pre></div></div>
<p>If you haven’t seen bytecode before (Dalvik or otherwise) don’t worry–most of it can be picked up without a full understanding.</p>
<p>In the first block, our <code class="highlighter-rouge">main</code> method, bytecode index <code class="highlighter-rouge">0000</code> retrieves a reference from a static <code class="highlighter-rouge">INSTANCE</code> field on a class named <code class="highlighter-rouge">Java8$1</code>. Since the original source didn’t contain a <code class="highlighter-rouge">Java8$1</code> class, we can infer that it was generated as part of desugaring. The <code class="highlighter-rouge">main</code> method’s bytecode also doesn’t contain any traces of the lambda body so it likely has to do with this <code class="highlighter-rouge">Java8$1</code> class. Index <code class="highlighter-rouge">0002</code> then calls the static <code class="highlighter-rouge">sayHi</code> method with the <code class="highlighter-rouge">INSTANCE</code> reference. The <code class="highlighter-rouge">sayHi</code> method requires a <code class="highlighter-rouge">Java8$Logger</code> argument so it would seem the <code class="highlighter-rouge">Java8$1</code> class implements that interface. We can verify all of this in the output.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Class #2 -
Class descriptor : 'LJava8$1;'
Access flags : 0x1011 (PUBLIC FINAL SYNTHETIC)
Superclass : 'Ljava/lang/Object;'
Interfaces -
#0 : 'LJava8$Logger;'
</code></pre></div></div>
<p>The presence of the <code class="highlighter-rouge">SYNTHETIC</code> flag means that the class was generated and the interfaces list includes <code class="highlighter-rouge">Java8$Logger</code>.</p>
<p>This class is now representing the lambda. If you look at its <code class="highlighter-rouge">log</code> method implementation, you might expect to find the missing lambda body.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>…
[00026c] Java8$1.log:(Ljava/lang/String;)V
0000: invoke-static {v1}, LJava8;.lambda$main$0:(Ljava/lang/String;)V
0003: return-void
…
</code></pre></div></div>
<p>Instead, it invokes a static method on the original <code class="highlighter-rouge">Java8</code> class named <code class="highlighter-rouge">lambda$main$0</code>. Again, the original source didn’t contain this method but it’s present in the bytecode.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>…
#1 : (in LJava8;)
name : 'lambda$main$0'
type : '(Ljava/lang/String;)V'
access : 0x1008 (STATIC SYNTHETIC)
[0002a0] Java8.lambda$main$0:(Ljava/lang/String;)V
0000: sget-object v0, Ljava/lang/System;.out:Ljava/io/PrintStream;
0002: invoke-virtual {v0, v1}, Ljava/io/PrintStream;.println:(Ljava/lang/String;)V
0005: return-void
</code></pre></div></div>
<p>The <code class="highlighter-rouge">SYNTHETIC</code> flag again confirms that this method was generated. And its bytecode contains the body of the lambda: a call to <code class="highlighter-rouge">System.out.println</code>. The reason that the lambda body is kept inside the original class is that it might access private members that the generated class wouldn’t have access to.</p>
<p>All of the puzzle pieces for understanding how desugaring works are here. Seeing it in Dalvik bytecode, though, can be a bit dense and intimidating.</p>
<h3 id="source-transformation">Source Transformation</h3>
<p>In order to better understand how desugaring works we can perform the transformation at the source code level. This is not how it actually works, but it’s a useful exercise for learning both what happens but also reinforcing what we saw in the bytecode.</p>
<p>Once again, we start from the original program with a lambda.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Java8</span> <span class="o">{</span>
<span class="kd">interface</span> <span class="nc">Logger</span> <span class="o">{</span>
<span class="kt">void</span> <span class="nf">log</span><span class="o">(</span><span class="nc">String</span> <span class="n">s</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">...</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
<span class="n">sayHi</span><span class="o">(</span><span class="n">s</span> <span class="o">-></span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">s</span><span class="o">));</span>
<span class="o">}</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">sayHi</span><span class="o">(</span><span class="nc">Logger</span> <span class="n">logger</span><span class="o">)</span> <span class="o">{</span>
<span class="n">logger</span><span class="o">.</span><span class="na">log</span><span class="o">(</span><span class="s">"Hello!"</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>First, the lambda body is moved to a sibling, package-private method.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> public static void main(String... args) {
<span class="gd">- sayHi(s -> System.out.println(s));
</span><span class="gi">+ sayHi(s -> lambda$main$0(s));
</span> }
<span class="gi">+
+ static void lambda$main$0(String s) {
+ System.out.println(s);
+ }
</span></code></pre></div></div>
<p>Then, a class is generated which implements the target interface and whose method body calls the lambda method.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> public static void main(String... args) {
<span class="gd">- sayHi(s -> lambda$main$0(s));
</span><span class="gi">+ sayHi(new Java8$1());
</span> }
<span class="err">@@</span>
}
<span class="gi">+
+class Java8$1 implements Java8.Logger {
+ @Override public void log(String s) {
+ Java8.lambda$main$0(s);
+ }
+}
</span></code></pre></div></div>
<p>Finally, because the lambda doesn’t capture any state, a singleton instance is created and stored in a static <code class="highlighter-rouge">INSTANCE</code> variable.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> public static void main(String... args) {
<span class="gd">- sayHi(new Java8$1());
</span><span class="gi">+ sayHi(Java8$1.INSTANCE);
</span> }
<span class="err">@@</span>
class Java8$1 implements Java8.Logger {
<span class="gi">+ static final Java8$1 INSTANCE = new Java8$1();
+
</span> @Override public void log(String s) {
</code></pre></div></div>
<p>This results in a fully desugared source file that can be used on all API levels.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Java8</span> <span class="o">{</span>
<span class="kd">interface</span> <span class="nc">Logger</span> <span class="o">{</span>
<span class="kt">void</span> <span class="nf">log</span><span class="o">(</span><span class="nc">String</span> <span class="n">s</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">...</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
<span class="n">sayHi</span><span class="o">(</span><span class="nc">Java8</span><span class="err">$</span><span class="mi">1</span><span class="o">.</span><span class="na">INSTANCE</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">static</span> <span class="kt">void</span> <span class="n">lambda$main</span><span class="err">$</span><span class="mi">0</span><span class="o">(</span><span class="nc">String</span> <span class="n">s</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">s</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">sayHi</span><span class="o">(</span><span class="nc">Logger</span> <span class="n">logger</span><span class="o">)</span> <span class="o">{</span>
<span class="n">logger</span><span class="o">.</span><span class="na">log</span><span class="o">(</span><span class="s">"Hello!"</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="kd">class</span> <span class="nc">Java8</span><span class="err">$</span><span class="mi">1</span> <span class="kd">implements</span> <span class="nc">Java8</span><span class="o">.</span><span class="na">Logger</span> <span class="o">{</span>
<span class="kd">static</span> <span class="kd">final</span> <span class="nc">Java8</span><span class="err">$</span><span class="mi">1</span> <span class="no">INSTANCE</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">Java8</span><span class="err">$</span><span class="mi">1</span><span class="o">();</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">log</span><span class="o">(</span><span class="nc">String</span> <span class="n">s</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">Java8</span><span class="o">.</span><span class="na">lambda</span><span class="n">$main</span><span class="err">$</span><span class="mi">0</span><span class="o">(</span><span class="n">s</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>If you actually look in the Dalvik bytecode for the generated lambda class it won’t have a name like <code class="highlighter-rouge">Java8$1</code>. The real name will look something like <code class="highlighter-rouge">-$$Lambda$Java8$QkyWJ8jlAksLjYziID4cZLvHwoY</code>. The reason for the awkward naming and the advantages it brings are content for another post…</p>
<h3 id="native-lambdas">Native Lambdas</h3>
<p>When we used the <code class="highlighter-rouge">dx</code> tool to attempt to compile lambda-containing Java bytecode to Dalvik bytecode its error message indicated that this would only work with a minimum API of 26 or newer.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ $ANDROID_HOME/build-tools/28.0.2/dx --dex --output . *.class
Uncaught translation error: com.android.dx.cf.code.SimException:
ERROR in Java8.main:([Ljava/lang/String;)V:
invalid opcode ba - invokedynamic requires --min-sdk-version >= 26
(currently 13)
1 error; aborting
</code></pre></div></div>
<p>Thus, if you re-run D8 and specify <code class="highlighter-rouge">--min-api 26</code> it’s reasonable to assume that “native” lambdas will be used and desugaring won’t actually occur.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ java -jar d8.jar \
--lib $ANDROID_HOME/platforms/android-28/android.jar \
--release \
--min-api 26 \
--output . \
*.class
</code></pre></div></div>
<p>But if you dump the .dex file, you’ll still find the <code class="highlighter-rouge">-$$Lambda$Java8$QkyWJ8jlAksLjYziID4cZLvHwoY</code> class was generated. Maybe it’s a D8 bug?</p>
<p>To learn why desugaring <em>always</em> occurs we need to look inside the Java bytecode of the <code class="highlighter-rouge">Java8</code> class.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ javap -v Java8.class
class Java8 {
public static void main(java.lang.String...);
Code:
0: invokedynamic #2, 0 // InvokeDynamic #0:log:()LJava8$Logger;
5: invokestatic #3 // Method sayHi:(LJava8$Logger;)V
8: return
}
…
</code></pre></div></div>
<p>The output has been trimmed for readability, but inside the <code class="highlighter-rouge">main</code> method you’ll see the <code class="highlighter-rouge">invokedynamic</code> bytecode at index <code class="highlighter-rouge">0</code>. The second argument to the bytecode is the value 0 which is the index of the associated bootstrap method. A bootstrap method is a bit of code that runs the first time that the bytecode is executed and it defines the behavior. The list of bootstrap methods are present at the bottom of the output.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>…
BootstrapMethods:
0: #27 invokestatic java/lang/invoke/LambdaMetafactory.metafactory:(
Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;
Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodType;
Ljava/lang/invoke/MethodHandle;Ljava/lang/invoke/MethodType;)
Ljava/lang/invoke/CallSite;
Method arguments:
#28 (Ljava/lang/String;)V
#29 invokestatic Java8.lambda$main$0:(Ljava/lang/String;)V
#28 (Ljava/lang/String;)V
</code></pre></div></div>
<p>In this case, the bootstrap method is called <code class="highlighter-rouge">metafactory</code> on the <code class="highlighter-rouge">java.lang.invoke.LambdaMetafactory</code> class. This class <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/invoke/LambdaMetafactory.html">lives in the JDK</a> and is responsible for creating anonymous classes on-the-fly at runtime for lambdas in a similar fashion to how D8 creates them at compile time.</p>
<p>If you look at the <a href="https://developer.android.com/reference/java/lang/invoke/package-summary">Android documentation for <code class="highlighter-rouge">java.lang.invoke</code></a> or the <a href="https://android.googlesource.com/platform/libcore/+/master/ojluni/src/main/java/java/lang/invoke/">AOSP source code for <code class="highlighter-rouge">java.lang.invoke</code></a>, though, you’ll notice this class isn’t present in the Android runtime. This is why desguaring always happens at compile-time regardless of your minimum API level. The VM has the bytecode support for an equivalent to <code class="highlighter-rouge">invokedynamic</code>, but the JDK’s built-in <code class="highlighter-rouge">LambdaMetafactory</code> is not available to use.</p>
<h3 id="method-references">Method References</h3>
<p>In addition to lambdas, method references were added to the language in Java 8. They’re an efficient way to create a lambda whose body points to an existing method.</p>
<p>The logger example in this post has been using a lambda body whose contents call an existing method, <code class="highlighter-rouge">System.out.println</code>. We can substitute the explicit lambda for a method reference to save some code.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> public static void main(String... args) {
<span class="gd">- sayHi(s -> System.out.println(s));
</span><span class="gi">+ sayHi(System.out::println);
</span> }
</code></pre></div></div>
<p>This compiles with <code class="highlighter-rouge">javac</code> and dexes with D8 the same as the lambda version with one notable difference. When dumping the Dalvik bytecode, the body of the generated lambda class has changed.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[000268] -$$Lambda$1Osqr2Z9OSwjseX_0FMQJcCG_uM.log:(Ljava/lang/String;)V
0000: iget-object v0, v1, L-$$Lambda$1Osqr2Z9OSwjseX_0FMQJcCG_uM;.f$0:Ljava/io/PrintStream;
0002: invoke-virtual {v0, v2}, Ljava/io/PrintStream;.println:(Ljava/lang/String;)V
0005: return-void
</code></pre></div></div>
<p>Instead of calling the generated <code class="highlighter-rouge">Java8.lambda$main$0</code> method which contains the call to <code class="highlighter-rouge">System.out.println</code>, the <code class="highlighter-rouge">log</code> implementation now invokes <code class="highlighter-rouge">System.out.println</code> directly.</p>
<p>The lambda class is also no longer a static singleton. Bytecode index <code class="highlighter-rouge">0000</code> above is reading an instance field for a <code class="highlighter-rouge">PrintStream</code> reference. This reference is <code class="highlighter-rouge">System.out</code> which is resolved at the call-site in <code class="highlighter-rouge">main</code> and passed into the constructor (which is named <code class="highlighter-rouge"><init></code> in bytecode).</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[0002bc] Java8.main:([Ljava/lang/String;)V
0000: sget-object v1, Ljava/lang/System;.out:Ljava/io/PrintStream;
0003: new-instance v0, L-$$Lambda$1Osqr2Z9OSwjseX_0FMQJcCG_uM;
0004: invoke-direct {v0, v1}, L-$$Lambda$1Osqr2Z9OSwjseX_0FMQJcCG_uM;.<init>:(Ljava/io/PrintStream;)V
0008: invoke-static {v0}, LJava8;.sayHi:(LJava8$Logger;)V
</code></pre></div></div>
<p>Performing the transformation at the source level again results in a straightforward transformation.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> public static void main(String... args) {
<span class="gd">- sayHi(System.out::println);
</span><span class="gi">+ sayHi(new -$$Lambda$1Osqr2Z9OSwjseX_0FMQJcCG_uM(System.out));
</span> }
<span class="err">@@</span>
}
<span class="gi">+
+class -$$Lambda$1Osqr2Z9OSwjseX_0FMQJcCG_uM implements Java8.Logger {
+ private final PrintStream ps;
+
+ -$$Lambda$1Osqr2Z9OSwjseX_0FMQJcCG_uM(PrintStream ps) {
+ this.ps = ps;
+ }
+
+ @Override public void log(String s) {
+ ps.println(s);
+ }
+}
</span></code></pre></div></div>
<h3 id="interface-methods">Interface Methods</h3>
<p>The other significant language feature of Java 8 was the ability to have <code class="highlighter-rouge">static</code> and <code class="highlighter-rouge">default</code> methods in interfaces. Static methods on interfaces allow providing instance factories or other helpers directly on the interface type on which they operate. Default methods allow you to compatibly add new methods to interfaces which have default implementations.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">interface</span> <span class="nc">Logger</span> <span class="o">{</span>
<span class="kt">void</span> <span class="nf">log</span><span class="o">(</span><span class="nc">String</span> <span class="n">s</span><span class="o">);</span>
<span class="k">default</span> <span class="kt">void</span> <span class="nf">log</span><span class="o">(</span><span class="nc">String</span> <span class="n">tag</span><span class="o">,</span> <span class="nc">String</span> <span class="n">s</span><span class="o">)</span> <span class="o">{</span>
<span class="n">log</span><span class="o">(</span><span class="n">tag</span> <span class="o">+</span> <span class="s">": "</span> <span class="o">+</span> <span class="n">s</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">static</span> <span class="nc">Logger</span> <span class="nf">systemOut</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">::</span><span class="n">println</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Both of these new method types on interfaces are supported by D8’s desugaring. Using the tools above it’s possible to understand how these are desugared to work on all API levels. That investigation is left as an exercise for the reader.</p>
<p>It is worth noting, though, that both of these features are implemented natively in the Android VM as of API 24. As a result, unlike lambdas and method references, specifying <code class="highlighter-rouge">--min-api 24</code> to D8 will result in them not having to be desugared.</p>
<h3 id="just-use-kotlin">Just Use Kotlin?</h3>
<p>By this point, a large majority of readers will have thought of Kotlin in some capacity. Yes, Kotlin provides lambdas and method references for passing code as data. Yes, Kotlin provides default and static(-like) functions on interfaces. All of those features are actually implemented by <code class="highlighter-rouge">kotlinc</code> in exactly the same way that D8 desugars the Java 8 bytecode (modulo small implementation details).</p>
<p>Android’s development toolchain and VM support of newer Java language features is still important even if you are writing 100% Kotlin code. New versions of Java bring more efficient constructs in both bytecode and in the VM that Kotlin can then take advantage of.</p>
<p>It’s not unreasonable to think that Kotlin will stop supporting Java 6 and Java 7 bytecode at some point in the future. The <a href="https://blog.jetbrains.com/idea/2015/12/intellij-idea-16-eap-144-2608-is-out/">IntelliJ platform has moved to Java 8</a> as of version 2016.1. Gradle 5.0 has moved to Java 8. The number of platforms running on older JVMs are dwindling. Without support for Java 8 bytecode and VM functionality, Android is in danger of becoming the largest ecosystem holding Kotlin’s Java bytecode generation back. Thankfully D8 and ART are stepping up here to ensure that isn’t the case.</p>
<h3 id="desugaring-apis">Desugaring APIs</h3>
<p>Thus far this post has focused on the language features and bytecode of newer Java versions. The other major benefit of new Java versions are the new APIs that come with it. Java 8 brought a ton of new APIs such as streams, <code class="highlighter-rouge">Optional</code>, functional interfaces, <code class="highlighter-rouge">CompletableFuture</code>, and a new date/time API.</p>
<p>Going back to the original logger example, we can use the new date/time API in order to know when messages were logged.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">java.time.*</span><span class="o">;</span>
<span class="kd">class</span> <span class="nc">Java8</span> <span class="o">{</span>
<span class="kd">interface</span> <span class="nc">Logger</span> <span class="o">{</span>
<span class="kt">void</span> <span class="nf">log</span><span class="o">(</span><span class="nc">LocalDateTime</span> <span class="n">time</span><span class="o">,</span> <span class="nc">String</span> <span class="n">s</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">...</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
<span class="n">sayHi</span><span class="o">((</span><span class="n">time</span><span class="o">,</span> <span class="n">s</span><span class="o">)</span> <span class="o">-></span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">time</span> <span class="o">+</span> <span class="s">" "</span> <span class="o">+</span> <span class="n">s</span><span class="o">));</span>
<span class="o">}</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">sayHi</span><span class="o">(</span><span class="nc">Logger</span> <span class="n">logger</span><span class="o">)</span> <span class="o">{</span>
<span class="n">logger</span><span class="o">.</span><span class="na">log</span><span class="o">(</span><span class="nc">LocalDateTime</span><span class="o">.</span><span class="na">now</span><span class="o">(),</span> <span class="s">"Hello!"</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>We can again compile this with <code class="highlighter-rouge">javac</code> and convert it to Dalvik bytecode with D8 which desugars it to run on all API levels.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ javac *.java
$ java -jar d8.jar \
--lib $ANDROID_HOME/platforms/android-28/android.jar \
--release \
--output . \
*.class
</code></pre></div></div>
<p>You can actually push this onto a phone or emulator to verify it works, something we didn’t do with the previous examples.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ adb push classes.dex /sdcard
classes.dex: 1 file pushed. 0.5 MB/s (1620 bytes in 0.003s)
$ adb shell dalvikvm -cp /sdcard/classes.dex Java8
2018-11-19T21:38:23.761 Hello
</code></pre></div></div>
<p>If your device runs API 26 or newer you will see a timestamp and the string “Hello!” as expected. But running it on a device with a version earlier than API 26 produces a very different result.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>java.lang.NoClassDefFoundError: Failed resolution of: Ljava/time/LocalDateTime;
at Java8.sayHi(Java8.java:13)
at Java8.main(Java8.java:9)
</code></pre></div></div>
<p>D8 has desugared the new language feature of lambdas to work on all API levels but didn’t do anything with the new API usage of <code class="highlighter-rouge">LocalDateTime</code>. This is disappointing because it means we only see <em>some</em> of the benefits of Java 8, not all of them.</p>
<p>Developers can choose to bundle their own <code class="highlighter-rouge">Optional</code> class or use a standalone version of the date/time library called <code class="highlighter-rouge">ThreeTenBP</code> to work around this. But if you can manually rewrite your code to use versions bundled in your APK, why can’t desugar in D8 do it for you?</p>
<p>It turns out that D8 already does this but only for a single API: <code class="highlighter-rouge">Throwable.addSuppressed</code>. This API is what allows the try-with-resources language feature of Java 7 to work on all versions of Android despite the API only being available from API 19.</p>
<p>All we need for the Java 8 APIs to work on all API levels then is a compatible implementation that we can bundle in the APK. It turns out the team that works on Bazel <a href="https://blog.bazel.build/2018/07/19/java-8-language-features-in-android-apps.html">have again already built this</a>. Their code that does the rewriting can’t be used, but the standalone repackaging of these JDK APIs can be. All we need is for the D8 team to add support in their desugaring tool to do the rewriting. You can <a href="https://issuetracker.google.com/issues/114481425">star the D8 feature request</a> on the Android issue tracker to convey your support.</p>
<hr />
<p>While the desugaring of language features has been available in various forms for some time, the lack of API desugaring remains a large gap in our ecosystem. Until the day that the majority of apps can specify a minimum API of 26, the lack of API desugaring in Android’s toolchain is holding back the Java library ecosystem. Libraries which support both Android and the JVM cannot use the Java 8 APIs that were introduced nearly 5 years ago!</p>
<p>And despite Java 8 language feature desugaring now being part of D8, it’s not enabled by default. Developers must explicitly opt-in by specifying their source and target compatibility to Java 8. Android library authors can help force this trend by building and publishing their libraries using Java 8 bytecode (even if you don’t use the language features).</p>
<p>D8 is being actively worked on and so the future still looks bright for Java language and API support. Even if you’re solely a Kotlin user, it’s important to maintain pressure on Android for support of new versions of Java for the better bytecodes and new APIs. And in some cases, D8 is actually ahead of the game for versions of Java beyond 8 which we’ll explore in the next post.</p>
<p><em>(This post was adapted from a part of my <a href="/digging-into-d8-and-r8">Digging into D8 and R8</a> talk that was never presented. Watch the video and look out for future blog posts for more content like this.)</em></p>
Increased accuracy of aapt2 "keep" rules2018-08-07T00:00:00+00:00https://jakewharton.com/increased-accuracy-of-aapt2-keep-rules<p>The <code class="highlighter-rouge">aapt2</code> tool packages your Android application resources into the format used at runtime. It also generates “keep” rules for ProGuard or R8 so that the types referenced inside of your resources do not get removed. Views referenced only in layout XML, action providers referenced only in menu XML, and broadcast receivers referenced only in the manifest XML are some examples of types that would otherwise be removed from the final APK were it not for these rules.</p>
<p>Prior to version 3.3.0-alpha05 of the Android Gradle plugin, <code class="highlighter-rouge">aapt2</code> would generate “keep” rules for the constructors of these types using an argument wildcard. Some rules for an application class, activity class, and view reference look like this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Referenced at frontend/android/build/intermediates/merged_manifests/release/AndroidManifest.xml:20
-keep class com.jakewharton.sdksearch.SdkSearchApplication { <init>(...); }
# Referenced at frontend/android/build/intermediates/merged_manifests/release/AndroidManifest.xml:28
-keep class com.jakewharton.sdksearch.ui.MainActivity { <init>(...); }
# Referenced at search/ui-android/build/intermediates/packaged_res/release/layout/search.xml:57
-keep class android.support.v7.widget.RecyclerView { <init>(...); }
</code></pre></div></div>
<p>Dumping the methods of the release APK we get:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>com.jakewharton.sdksearch.SdkSearchApplication <init>()
com.jakewharton.sdksearch.ui.MainActivity <init>()
android.support.v7.widget.RecyclerView <init>(Context)
android.support.v7.widget.RecyclerView <init>(Context, AttributeSet)
android.support.v7.widget.RecyclerView <init>(Context, AttributeSet, int)
</code></pre></div></div>
<p><code class="highlighter-rouge">SdkSearchApplication</code> and <code class="highlighter-rouge">MainActivity</code> contain only a default constructor but <code class="highlighter-rouge">RecyclerView</code> contains three. As far as the reflective lookup is concerned, only one constructor will be used. For types in the manifest the default (no-argument) constructor is used. For types in a layout XML file the two-arg <code class="highlighter-rouge">Context</code>+<code class="highlighter-rouge">AttributeSet</code> constructor is invoked by <code class="highlighter-rouge">LayoutInflater</code>. By generating rules with <code class="highlighter-rouge"><init>(...)</code> we are forcing every constructor to be retained despite only needing one.</p>
<p>Starting with version 3.3.0-alpha05 of the Android Gradle plugin, a new version of <code class="highlighter-rouge">aapt2</code> is used which generates more precise rules that reference only the exact constructor which the reflective lookup will use.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Referenced at frontend/android/build/intermediates/merged_manifests/release/AndroidManifest.xml:20
-keep class com.jakewharton.sdksearch.SdkSearchApplication { <init>(); }
# Referenced at frontend/android/build/intermediates/merged_manifests/release/AndroidManifest.xml:28
-keep class com.jakewharton.sdksearch.ui.MainActivity { <init>(); }
# Referenced at search/ui-android/build/intermediates/packaged_res/release/layout/search.xml:57
-keep class android.support.v7.widget.RecyclerView { <init>(android.content.Context, android.util.AttributeSet); }
</code></pre></div></div>
<p>Dumping the methods of the release APK again now shows:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>com.jakewharton.sdksearch.SdkSearchApplication <init>()
com.jakewharton.sdksearch.ui.MainActivity <init>()
android.support.v7.widget.RecyclerView <init>(Context, AttributeSet)
android.support.v7.widget.RecyclerView <init>(Context, AttributeSet, int)
</code></pre></div></div>
<p>The <code class="highlighter-rouge"><init>(Context)</code> of <code class="highlighter-rouge">RecyclerView</code> is no longer present! That constructor used to be forced into the release APK despite never actually being used. The three-argument constructor is still kept is because the two-argument one delegates to it:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="nf">RecyclerView</span><span class="o">(</span><span class="nd">@NonNull</span> <span class="nc">Context</span> <span class="n">context</span><span class="o">,</span> <span class="nd">@Nullable</span> <span class="nc">AttributeSet</span> <span class="n">attrs</span><span class="o">)</span> <span class="o">{</span>
<span class="k">this</span><span class="o">(</span><span class="n">context</span><span class="o">,</span> <span class="n">attrs</span><span class="o">,</span> <span class="mi">0</span><span class="o">);</span>
<span class="o">}</span>
</code></pre></div></div>
<p>If optimization is also enabled and there are no other uses of that three-argument constructor it may get inlined–something that couldn’t have happened with the old rules.</p>
<p>This seems like a small change, and it mostly is. Application, activity, and action provider subtypes tend to only have the one constructor so their counts are unlikely to change. View subtypes, however, very frequently have three or four constructors and you will likely now see two or three of those being removed. In the scope of an entire APK that allows on the order of tens or hundreds of methods to be removed which were <em>needlessly</em> being kept. As the specificity of “keep” rules increases it not only reduces the raw number of methods that wind up in the final APK, but often allows optimization passes to have a greater effect.</p>
<p>If you find any bugs with the new rules, please report them on the <a href="https://issuetracker.google.com/issues/new?component=192709">Android issue tracker</a>.</p>
Tracing Gradle task execution2018-08-01T00:00:00+00:00https://jakewharton.com/tracing-gradle-task-execution<p>Gradle provides two built-in mechanisms for tracing your build: <code class="highlighter-rouge">--profile</code> and <code class="highlighter-rouge">--scan</code>. The former produces a simple HTML report of task execution times. You can get a rough idea of where time was spent but are unlikely to glean any real insights. The latter sends a detailed report to Gradle’s servers (or to a Gradle Enterprise installation) with much more granular information. Task details are rendered on a concurrent timeline corresponding to their execution. For CI builds, I tend to want something more granular than <code class="highlighter-rouge">--profile</code> but I don’t like the idea of sending details of every build to Gradle with <code class="highlighter-rouge">--scan</code>. It seems entirely needless considering their plugin has all of that information locally but chooses to render it remotely.</p>
<p>The <a href="https://github.com/gradle/gradle-profiler/">Gradle profiler</a> project started a few years ago as a way to deterministically measure build speeds. By creating scenarios such as an ABI-breaking change, ABI-compatible change, Android resource change, etc., the tool can run these scenarios multiple times to first warm up the JVM and then to produce an accurate picture of what gets executed. It offers integrations and outputs for use with popular JVM-based performance analysis tools such as YourKit and Java Flight Recorder.</p>
<p>For CI builds, executing through the Gradle profiler would be an annoying abstraction to use. We can instead use it for inspiration and run its integrations on individual builds.</p>
<hr />
<p>Java Flight Recorder can be used on individual Gradle builds with the <code class="highlighter-rouge">jcmd</code> binary in the JDK and with flags to <code class="highlighter-rouge">java</code> specified on the <code class="highlighter-rouge">org.gradle.jvmargs</code> in your <code class="highlighter-rouge">gradle.properties</code>. There are even <a href="https://github.com/lhotari/jfr-gradle-plugin">Gradle plugins</a> which offer to start and stop the recording automatically. We can then open the resulting <code class="highlighter-rouge">.jfr</code> file in Java Mission Control or use <a href="https://github.com/lhotari/jfr-report-tool">a command-line tool</a> to convert it into a <a href="http://www.brendangregg.com/flamegraphs.html">flamegraph</a>.</p>
<p><a href="/static/post-image/trace-flame.png"><img src="/static/post-image/trace-flame.png" alt="Flame graph of SDK Search build" /></a></p>
<p>The flamegraph can show where time is being spent inside of tasks over the course of the build. The stacks aren’t correlated to a task, though, so it’s important to remember that you’re looking at the larger picture. This also doesn’t handle tasks which communicate with their own daemons such as the Kotlin compiler.</p>
<p>While this produces a pretty output, its utility is small and the Gradle plugin integration is not the most stable. I would refrain from using this on CI as result unless you’re going to build out a strong integration with <code class="highlighter-rouge">jcmd</code> directly. These visualizations work well when you have a small subset of tasks to run rather than when your entire project is being built.</p>
<hr />
<p>The Gradle profiler also includes support for <a href="https://www.chromium.org/developers/how-tos/trace-event-profiling-tool">Chrome traces</a>. This output will be familiar to Android users who have used the <a href="https://developer.android.com/studio/command-line/systrace">systrace tool</a>. Again we can integrate this into our builds without jumping through the Gradle profiler.</p>
<p>The code for producing a Chrome trace lives inside <a href="https://github.com/gradle/gradle-profiler/">the Gradle profiler repository</a>. Clone and build the project which will produce a jar at <code class="highlighter-rouge">subprojects/chrome-trace/build/libs/chrome-trace.jar</code>. Copy this jar into the <code class="highlighter-rouge">gradle/</code> directory of your project. This jar contains a plugin which can be applied inside a Gradle initialization script.</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// init.gradle</span>
<span class="n">initscript</span> <span class="o">{</span>
<span class="n">dependencies</span> <span class="o">{</span>
<span class="n">classpath</span> <span class="nf">files</span><span class="o">(</span><span class="s1">'gradle/chrome-trace.jar'</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="n">rootProject</span> <span class="o">{</span>
<span class="kt">def</span> <span class="n">date</span> <span class="o">=</span> <span class="k">new</span> <span class="n">java</span><span class="o">.</span><span class="na">text</span><span class="o">.</span><span class="na">SimpleDateFormat</span><span class="o">(</span><span class="s2">"yyyy-MM-dd-HH-mm-ss"</span><span class="o">).</span><span class="na">format</span><span class="o">(</span><span class="k">new</span> <span class="n">Date</span><span class="o">())</span>
<span class="n">ext</span><span class="o">.</span><span class="na">chromeTraceFile</span> <span class="o">=</span> <span class="k">new</span> <span class="n">File</span><span class="o">(</span><span class="n">rootProject</span><span class="o">.</span><span class="na">buildDir</span><span class="o">,</span> <span class="s2">"reports/trace/trace-${date}.html"</span><span class="o">)</span>
<span class="o">}</span>
<span class="n">apply</span> <span class="nl">plugin:</span> <span class="n">org</span><span class="o">.</span><span class="na">gradle</span><span class="o">.</span><span class="na">trace</span><span class="o">.</span><span class="na">GradleTracingPlugin</span>
</code></pre></div></div>
<p>When invoking Gradle we need to reference this script and also pass a flag to enable the tracing.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ ./gradlew --init-script init.gradle -Dtrace build
</code></pre></div></div>
<p>This will produce a trace file at <code class="highlighter-rouge">build/reports/trace/trace-(date).html</code> which you can open in Chrome and navigate using the arrow keys and A-S-D-W keys.</p>
<p><a href="/static/post-image/trace-1.png"><img src="/static/post-image/trace-1.png" alt="Chrome trace of SDK Search build" /></a></p>
<p>The trace gives a picture of concurrent task execution and timings therein. There is very little here that isn’t in the <code class="highlighter-rouge">--profile</code> report, but it’s presented in a manner that gives you more context. The most notable and welcome addition is that of CPU load, heap size, and GC events.</p>
<p>Unfortunately, the granularity per-task is near zero. There are no insights into <a href="https://guides.gradle.org/using-the-worker-api/">workers</a> that operate as part of a task. We cannot get flame graphs of the call stacks inside of a task.</p>
<p>I have added this to SDK Search’s CI builds in addition to the other reports it already generates if you’d like to see a full integration: <a href="https://github.com/JakeWharton/SdkSearch/commit/3cc9bd8bc9741cf8459bf975a186e0c36e5481d8">https://github.com/JakeWharton/SdkSearch/commit/3cc9bd8bc9741cf8459bf975a186e0c36e5481d8</a>.</p>
<hr />
<p>Neither is perfect but both can be useful in different situations. Hopefully in the future visibility into workers will be added to the Chrome trace. Figuring out how to merge the Java Flight Recorder data into the Chrome trace would also be an amazing addition. For now, having the Chrome trace run on CI gives a good picture of how the build is performing and then Java Flight Recorder can be used either manually or with the Gradle profiler to dig into individual task performance.</p>
<p>Here are the four tracing outputs of a single build:</p>
<ul>
<li><a href="/static/files/trace/profile.html"><code class="highlighter-rouge">--profile</code> report</a></li>
<li><a href="/static/files/trace/trace.html">Chrome trace</a></li>
<li><a href="/static/files/trace/jfr.svg">JFR flamegraph</a></li>
<li><a href="https://gradle.com/s/xtvvyrmkuwobe"><code class="highlighter-rouge">--scan</code> report</a></li>
</ul>
Introducing Android KTX: Even Sweeter Kotlin Development for Android2018-02-05T00:00:00+00:00https://jakewharton.com/introducing-android-ktxThis post was published externally on Android Developers Blog. Read it at https://android-developers.googleblog.com/2018/02/introducing-android-ktx-even-sweeter.html.Surfacing Hidden Change to Pull Requests2017-07-13T00:00:00+00:00https://jakewharton.com/surfacing-hidden-change-to-pull-requestsThis post was published externally on Square Corner. Read it at https://developer.squareup.com/blog/surfacing-hidden-change-to-pull-requests.Generating Kotlin code with KotinPoet2017-05-16T00:00:00+00:00https://jakewharton.com/generating-kotlin-code-with-kotlinpoetThis post was published externally on Square Corner. Read it at https://developer.squareup.com/blog/generating-kotlin-code-with-kotlinpoet/.An Optional's place in Kotlin2017-05-14T00:00:00+00:00https://jakewharton.com/an-optionals-place-in-kotlinThis post was published externally on Square Corner. Read it at https://developer.squareup.com/blog/an-optionals-place-in-kotlin.Square Open Source ♥s Kotlin2017-05-12T00:00:00+00:00https://jakewharton.com/square-open-source-loves-kotlinThis post was published externally on Square Corner. Read it at https://developer.squareup.com/blog/square-open-source-loves-kotlin.Web Sockets now shipping in OkHttp 3.5!2016-12-02T00:00:00+00:00https://jakewharton.com/web-sockets-now-shipping-in-okhttpThis post was published externally on Square Corner. Read it at https://developer.squareup.com/blog/web-sockets-now-shipping-in-okhttp-3-5.Forcing bytes downward in Okio2016-09-06T00:00:00+00:00https://jakewharton.com/forcing-bytes-downward-in-okio<p>Okio’s <code class="highlighter-rouge">BufferedSink</code> is a high-level abstraction for writing binary and character data as bytes.
Its design stems from frustrations with the JDK’s <code class="highlighter-rouge">java.io.*</code> and <code class="highlighter-rouge">java.nio.*</code> libraries. At
Droidcon Montreal last year I gave <a href="/a-few-ok-libraries">a presentation</a> comparing
it with the former, but also showcased Okio’s concept of a <em>segment</em> and how it enables the library
to cheaply move bytes. If you aren’t familiar with Okio I encourage you to go watch the presentation
first since the rest of this post will assume at least cursory knowledge of its types.</p>
<p>If you never look at the source code of Okio you won’t know that this segment concept exists.
It’s an implementation detail for performance that remains completely opaque to the consumer of the
library. That is, except for one<sup>*</sup> notable exception: the <code class="highlighter-rouge">emitCompleteSegments()</code> method
on <code class="highlighter-rouge">BufferedSink</code>.</p>
<p>This method is part of a family of three methods which force buffered bytes to be moved to the
underlying <code class="highlighter-rouge">Sink</code>. Their difference is subtle, but understanding that difference ensures correctness
and can make or break throughput.</p>
<p>First let’s understand the difference in behavior and then look at some use cases for each.</p>
<hr />
<p><strong><code class="highlighter-rouge">flush()</code></strong></p>
<p><img src="/static/post-image/okio-flush.gif" height="240" width="210" class="float-right" /></p>
<p>Flush is a common concept in stream APIs and its semantics remain unchanged in Okio. Calls to this
method cause <em>all buffered bytes</em> to be moved to the underlying <code class="highlighter-rouge">Sink</code> and then that <code class="highlighter-rouge">Sink</code>
is also instructed to flush itself. When calls to <code class="highlighter-rouge">flush()</code> return, you are guaranteed that all
bytes have been sent all the way to the destination <code class="highlighter-rouge">Sink</code>.</p>
<p>When multiple levels of buffering are in use, a call to <code class="highlighter-rouge">flush()</code> will clear the buffers at every
level. In Okio multiple levels of buffering are so cheap that it’s practically free. A flush just
amounts to each level <em>moving</em> its segments down to the next level of the chain.</p>
<p>With <code class="highlighter-rouge">java.io.*</code> streams, however, multiple levels of buffering require each level to allocate
and manage its own <code class="highlighter-rouge">byte[]</code>. This means that a flush operation will result in each level doing an
<code class="highlighter-rouge">arraycopy()</code> of its data down to the next level (which also might have required buffer expansion).</p>
<p>Calls to <code class="highlighter-rouge">close()</code> on stream types typically behave similar to <code class="highlighter-rouge">flush()</code> in that they write all
buffered bytes to the underlying stream before also instructing it to close.</p>
<p><strong><code class="highlighter-rouge">emit()</code></strong></p>
<p><img src="/static/post-image/okio-emit.gif" height="240" width="210" class="float-right" /></p>
<p>Emitting bytes is very similar to flushing except that it is not a recursive operation. Calls to
this method cause all buffered bytes to be moved to the underlying <code class="highlighter-rouge">Sink</code>. Unlike <code class="highlighter-rouge">flush()</code>,
however, that <code class="highlighter-rouge">Sink</code> is not told to do any other operations.</p>
<p>Because buffering is so inexpensive with Okio, it’s not uncommon to accept a <code class="highlighter-rouge">Sink</code> for API flexibility and
immediately wrap it in a <code class="highlighter-rouge">BufferedSink</code> for the implementation’s convenience. It’s important to not
leave any buffered bytes unwritten to the original <code class="highlighter-rouge">Sink</code> which is what calls to <code class="highlighter-rouge">emit()</code> will ensure.</p>
<p><code class="highlighter-rouge">emit()</code> is a nice alternative to <code class="highlighter-rouge">flush()</code> since it allows you to use the more useful
<code class="highlighter-rouge">BufferedSink</code> type without the concern of needlessly causing bytes to be sent all the way down the
chain every time you’re finished with the abstraction.</p>
<p><strong><code class="highlighter-rouge">emitCompleteSegments()</code></strong></p>
<p><img src="/static/post-image/okio-emit-complete-segments.gif" height="240" width="210" class="float-right" /></p>
<p>If you understand the behavior of <code class="highlighter-rouge">emit()</code> and you understood the concept of segments then the
behavior of this method should be straightforward. Calls to this method cause only the bytes which
are part of <em>complete segments</em> to be moved to the underlying <code class="highlighter-rouge">Sink</code>. If you haven’t buffered
enough bytes to create a complete segment this method will actually do nothing!</p>
<p>Remember, segments are an implementation detail of Okio and as such so are their sizes. So why does
the public API expose their concept in this method?</p>
<p>The reason this method exists to ensure that your code is actually not buffering <em>too many bytes</em>.
When sending large amounts of data over a long period of time through a <code class="highlighter-rouge">BufferedSink</code>, it can be
beneficial to occasionally write parts of the data to the underlying <code class="highlighter-rouge">Sink</code>. This ensures that the
destination isn’t overwhelmed by a single gigantic write. Instead, the <code class="highlighter-rouge">Sink</code> can incrementally
process bytes as they’re available and optionally send back signals to the producer (either with an
exception or out-of-band notifications like HTTP/2’s flow control).</p>
<hr />
<p>Now that we know the difference in behavior of these three methods, let’s look at some use cases
of when you would want to use each.</p>
<p><strong>Writing messages to a WebSocket</strong></p>
<p>WebSockets are long-lived connections to a server over which string or binary messages are
constantly streamed in both directions. The frequency of these messages can be extremely rapid or
quite sparse. An API for sending messages on a <code class="highlighter-rouge">WebSocket</code> class would look something like this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>public void sendMessage(String content);
public void sendMessage(byte[] content);
</code></pre></div></div>
<p>This <code class="highlighter-rouge">WebSocket</code> type would wrap a socket stream with a <code class="highlighter-rouge">BufferedSink</code> for sending messages. Because
we don’t know when the next message will come from the application, the implementation of
<code class="highlighter-rouge">sendMessage</code> needs to call <code class="highlighter-rouge">flush()</code> before returning to optimize for latency. This ensures all of
the data from each message will be sent down through the socket.</p>
<p>If you were to use <code class="highlighter-rouge">emitCompleteSegments()</code> part of the message would almost always be left in the
buffer. Using <code class="highlighter-rouge">emit()</code> would only work if there’s no intermediate buffering which is hard to
guarantee. This is why <code class="highlighter-rouge">flush()</code> is the only appropriate operation for this example.</p>
<p><strong>Encoding a video to a file</strong></p>
<p>Video encoding is a CPU-bound, memory-intensive process which generates data at a fairly consistent
rate. Writing this data to a file as its being encoded keeps the buffer size small and ensures that
the slower disk drive can keep up.</p>
<p>At regular intervals the implementation writing encoded data to a <code class="highlighter-rouge">BufferedSink</code> should call
<code class="highlighter-rouge">emitCompleteSegments()</code> to allow large portions of the buffer to get moved to the underlying
<code class="highlighter-rouge">Sink</code> and start trickling down the chain. The reason that <code class="highlighter-rouge">emitCompleteSegments()</code> is preferred
here over <code class="highlighter-rouge">emit()</code> is that more data will be coming into buffer. Sending a partially-completed
segment would be wasteful since it has empty bytes that can still be used for the incoming data.</p>
<p>It’s important to note again that <code class="highlighter-rouge">emitCompleteSegments()</code> only writes to the underlying <code class="highlighter-rouge">Sink</code>
and not all the way down the chain (i.e., it’s not <code class="highlighter-rouge">flushCompleteSegments()</code>). This means that if
there is an intermediate buffer which isn’t monitoring its size and occasionally calling this method
you will end up buffering the whole video.</p>
<p>When the video is done encoding and no more bytes will be written a call to <code class="highlighter-rouge">emit()</code> (with the same
caveats as the last paragraph) or <code class="highlighter-rouge">flush()</code> should happen so that the final bytes are not left in
the buffer.</p>
<p><strong>Serializing an object to JSON</strong></p>
<p>If we wanted to create a library that took an object and serialized it to JSON we would probably
give it a method signature like this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>public void toJson(Object o, Sink sink)
</code></pre></div></div>
<p>Because <code class="highlighter-rouge">Sink</code> offers no convenience on its own, the implementation would buffer it into an
<code class="highlighter-rouge">BufferedSink</code> for access to high-level APIs.</p>
<p>Once the implementation was finished writing the JSON representation to the <code class="highlighter-rouge">BufferedSink</code> it needs
to return. In order to not leave any bytes in the buffer it needs to call <code class="highlighter-rouge">emit()</code>. This writes all
of the buffered bytes to the underlying <code class="highlighter-rouge">Sink</code>, but not any further. Whether or not the <code class="highlighter-rouge">Sink</code>
should be flushed all the way down is left as a decision for the caller. This affords more control
so that if the caller wants to serialize multiple objects and flush them all at once they are able
to do so.</p>
<p>Astute readers might be aware of <a href="https://github.com/square/moshi">Moshi</a>–a small JSON
serialization library built with Okio. Unlike our hypothetical <code class="highlighter-rouge">toJson</code> method above, Moshi’s method
actually requires a <code class="highlighter-rouge">BufferedSink</code> directly. The reason for this is completely unrelated to flushing
or emitting data, but rather for symmetry with the <code class="highlighter-rouge">fromJson</code> method which requires a
<code class="highlighter-rouge">BufferedSource</code>.</p>
<hr />
<p>So <code class="highlighter-rouge">flush()</code>, <code class="highlighter-rouge">emit()</code>, and <code class="highlighter-rouge">emitCompleteSegments()</code> each instruct a <code class="highlighter-rouge">BufferedSink</code> to move data to
the underlying <code class="highlighter-rouge">Sink</code> in slightly different ways. Understanding that difference ensures that your
bytes do not get lost inside of intermediate buffers but also that you move the minimal amount of
bytes for your needs.</p>
<p> </p>
<p> </p>
<p> </p>
<p><em><small><sup>*</sup> Technically there is two exceptions. <code class="highlighter-rouge">Buffer</code> has a <code class="highlighter-rouge">completeSegmentByteCount()</code> method which returns the number of bytes that would be moved by a call to <code class="highlighter-rouge">emitCompleteSegments()</code>.</small></em></p>
Just Say mNo to Hungarian Notation2016-01-21T00:00:00+00:00https://jakewharton.com/just-say-no-to-hungarian-notation<p>Every day new Java code is written for Android apps and libraries which is plagued with an infectious disease: Hungarian notation.</p>
<p>The proliferation of Hungarian notation on Android is an accident and its continued justification erroneous. Let’s dispel its common means of advocacy:</p>
<ul>
<li>
<p>“<strong>The Android Java style guide recommends its use</strong>”</p>
<p>There is no such thing as an Android Java style guide that provides any guidance on how <em>you</em> should write Java code. Most people referencing this non-existent style guide are referring to the <a href="http://s.android.com/source/code-style.html#follow-field-naming-conventions">style guide for contributions to the Android Open Source Project (AOSP)</a>.</p>
<p>You are not writing code for AOSP so you do not need to follow their style guide.</p>
<p>If you’re working on code that might someday live in AOSP you don’t even need to follow this style guide. Almost all of the Java libraries imported by AOSP do not follow it, and even some of the ones developed inside of AOSP don’t either.</p>
</li>
<li>
<p>“<strong>The Android samples use it</strong>”</p>
<p>These samples started life in the platform inside of AOSP so they adhere to the AOSP style. For those which did not come from AOSP, the author either incorrectly believes the other points of advocation in this post or simply forget to <em>correct</em> their style when writing the sample.</p>
</li>
<li>
<p>“<strong>The extra information helps in code review</strong>”</p>
<p>The ‘m’ or ‘s’ prefix on name indicates a private/package instance field or private/package static field, respectively, where this would otherwise not be known in code review. This assumes the field isn’t visible in the change, since then its visibility would obviously be known regardless.</p>
<p>Before I attempt to refute this, let’s define Hungarian notation. <a href="https://en.wikipedia.org/wiki/Hungarian_notation#Systems_vs._Apps_Hungarian">According to Wikipedia</a>, there are two types of Hungarian notations:</p>
<ul>
<li>System notation encoded the data type of the variable in its name. A user ID that was a <code class="highlighter-rouge">long</code> represented in Java would name a variable <code class="highlighter-rouge">lUserId</code> to indicate both usage and type information.</li>
<li>Apps notation encoded the semantic use of the variable rather than it’s logical use or purpose. A variable for storing private information had a prefix (like <code class="highlighter-rouge">mUserId</code>) whereas a variable for storing public information had another prefix, or none whatsoever.</li>
</ul>
<p>So when you see the usage of a field, which piece of information is more important for the review: the visibility of that field or the type of that field?</p>
<p>The visibility is a <em>useless</em> attribute to care about in a code review. The field is already present and available for use, and presumably its visibility was code-reviewed in a previous change. The type of a field, however, has a direct impact on <em>how</em> that field can being used in the change. The correct methods to call, the position in arguments, and the methods which can be called all are directly related to its type.</p>
<p>Not only is advocating for ‘apps’ Hungarian wrong because it’s not useful, but it’s doubly wrong since ‘system’ Hungarian would provide more relevant info. That’s not to say you should use ‘system’, both the type and visibility of a field changes and you will forget to update the name. It’s not hard to find <a href="https://github.com/square/leakcanary/blob/4950e1756c79fba871f524d5d9c47ed9322b23b3/leakcanary-android/src/main/java/com/squareup/leakcanary/AndroidExcludedRefs.java#L263-L269">static <code class="highlighter-rouge">mContext</code> fields</a>, after all.</p>
</li>
<li>
<p>“<strong>The extra information helps in development</strong>”</p>
<p>Android Studio and IntelliJ IDEA visually distinguish field names based on membership (instance or static):</p>
<p><img src="/static/post-image/hungarian-idea.png" width="283" /></p>
<p>IDEs will enforce correct membership, visibility, and types by default so a naming convention isn’t going to add anything here. A popup showing all three properties (and more) of a field is also just a keypress away.</p>
</li>
<li>
<p>“<strong>I want to write Java code like Google does</strong>”</p>
<p>While Android and AOSP are part of the company, Google explicitly and actively forbids Hungarian notation <a href="https://google.github.io/styleguide/javaguide.html#s5.1-identifier-names">in their Java style guide</a>. This public Java style guideline is the formalization of long-standing internal conventions.</p>
<p>Android had originated outside of Google and the team early on chose to host the Hungarian disease. Changing it at this point would be needless churn and cause many conflicts across branches and third-party partners.</p>
</li>
</ul>
<p>With your continued support and activism on this topic, this disease can be eradicated in our lifetime.</p>
<p><code class="highlighter-rouge">mFriends</code> don’t let <code class="highlighter-rouge">sFriends</code> use Hungarian notation!</p>
Java Interoperability Policy for Major Version Updates2015-12-11T00:00:00+00:00https://jakewharton.com/java-interoperability-policy-for-major-version-updates<p>Major version updates to libraries solve the API warts of old and bring shiny new APIs to address previous shortcomings—often in a breaking fashion. Updating an Android or Java app is usually a day or two affair before you reap the benefits. Problems arise, however, when other libraries you depend on have transitive dependencies on older versions of the updated library.</p>
<p><a href="https://github.com/square/retrofit/">Retrofit 2.0</a> is nearing release and it comes with three years of knowledge gained since its version 1.0—some of which is in backwards-incompatible API changes. We are fortunate to say that Retrofit has become a popular library, but it presents a real problem in that other libraries have been published which rely on its 1.x API. While a sudden breaking change doesn’t present an immediate problem for them, consumers of those libraries wanting to upgrade their apps to the new API face a difficult choice.</p>
<p>This problem is not new, and I won’t waste time rehashing all its nuances. After some discussion with <a href="https://twitter.com/jessewilson">Jesse Wilson</a>, we have decided on a course of action for the libraries we manage going forward in order to mitigate this pain. The following does not assume strict semantic versioning, but general adherence to its idea of major version bumps.</p>
<p>For major version updates in significantly foundational libraries we will take the following steps:</p>
<ol>
<li>
<p><strong>Rename the Java package to include the version number.</strong></p>
<p>This immediately solves the API compatibility problem from transitive dependencies on multiple versions. Classes from each can be loaded on the same classpath without interacting negatively.</p>
<p>Users can perform major versions updates gradually or in increments rather than requiring an immediate switch. If possible, shims for older versions can be built on newer versions in a sibling artifact.</p>
<p>For example, versions 0.x and 1.x would be under <code class="highlighter-rouge">com.example.retrofit</code>, versions 2.x would be under <code class="highlighter-rouge">com.example.retrofit2</code>, and so on.</p>
<p>(Libraries with a major version of 0 or 1 can skip this, and only start with major version 2 and above.)</p>
</li>
<li>
<p><strong>Include the library name as part the group ID in the Maven coordinates.</strong></p>
<p>Even for projects that have only a single artifact, including the project name in the group ID allows future updates that may introduce additional artifacts to not pollute the root namespace. In projects that have multiple artifacts from inception, it provides a means of grouping them together on artifact hosts like Maven central.</p>
<p>For example, the Maven coordinates for the main Retrofit artifact could be <code class="highlighter-rouge">com.example.retrofit:retrofit</code>. Additional modules (present or future) can be listed under the same group ID such as <code class="highlighter-rouge">com.example.retrofit:converter-moshi</code>.</p>
</li>
<li>
<p><strong>Rename the group ID in the Maven coordinates to include the version number.</strong></p>
<p>Individual group IDs prevent dependency resolution semantics to upgrade older versions to newer, incompatible ones. Each major version is resolved independently allowing transitive dependencies to be upgraded compatibly.</p>
<p>For example, take a project given <em>Library A</em> with a dependency on 1.2.0, <em>Library B</em> with a dependency on 1.3.0, <em>Library C</em> with a dependency on 2.1.0, and a direct dependency on 2.4.0. A dependency resolver would first choose 1.3.0 for which <em>Library A</em> and <em>Library B</em> are compatible using the 1.x group ID. The resolver would then choose 2.4.0 for which <em>Library C</em> is compatible using the 2.x group ID.</p>
<p>Group ID renaming is chosen over the artifact ID for a few reasons:</p>
<ul>
<li>The filename of built artifacts is the combination of the artifact ID the the version. If the artifact ID contained the major version it would appear redundant (e.g., <code class="highlighter-rouge">retrofit2-2.1.0</code>).</li>
<li>Projects can be comprised of multiple artifacts and not all of them contain the raw name of the project. Properly describing the contents of the artifact is more important than including versioning information.</li>
<li>Maven-based builds reference dependencies on sibling modules in the same project using their artifact ID but can use variables for the group ID and version. If the artifact ID were to change, a lot of error-prone <code class="highlighter-rouge">pom.xml</code> changes would be required instead of one group ID change.</li>
</ul>
<p>(Libraries with a major version of 0 or 1 can skip this, and only start with major version 2 and above.)</p>
</li>
</ol>
<hr />
<p>Each of these steps are not new ideas themselves. The growing usage of the libraries on which we work has forced us to figure out a reasonable policy to ensure major version upgrades are as smooth as possible. We are excited to offer something that will allow our users to upgrade sooner while also having relatively low maintenance cost for us.</p>
<p>The forthcoming releases of Retrofit 2.0 and OkHttp 3.0 will be the first two libraries to apply this policy. Enjoy!</p>
SQLBrite: A reactive Database Foundation2015-02-25T00:00:00+00:00https://jakewharton.com/sqlbrite-a-reactive-database-foundationThis post was published externally on Square Corner. Read it at https://developer.squareup.com/blog/sqlbrite-a-reactive-database-foundation.Better Parameterized Tests with Burst2014-11-21T00:00:00+00:00https://jakewharton.com/better-parameterized-tests-with-burstThis post was published externally on Square Corner. Read it at https://developer.squareup.com/blog/better-parameterized-tests-with-burst.The Conference Speaker Investment2014-11-20T00:00:00+00:00https://jakewharton.com/the-conference-speaker-investment<p>Speaking at conferences is an investment of speakers in time, energy, and knowledge. The quality of a presentation and of the conference itself can be measured in the amount of investment made.</p>
<p>Here are my tips for future conference speakers.</p>
<hr />
<h3 id="choosing-a-conference">Choosing A Conference</h3>
<p>At which conference you choose to speak is arguably as important as the topic selection and the amount of work put into the presentation. There are very obvious factors that everyone should consider such as the overall theme, audience that will be attending, and overall conference size.</p>
<p>A conference of 200 people on a specific topic in a remote region deserves a very different preparation than one of 2000 people on a general topic in a large tech hub. The smaller of these two would require something very detailed on which you consider yourself an expert because the people in attendance will expect it. At the larger one you can still give the same detailed talk, but you also have the option of being more broad, appealing more to introductory learners, and not having to be an absolute authority on the subject.</p>
<p>Beyond those, there are other very important factors that I consider essential criteria when selecting where to speak.</p>
<h4 id="recording">Recording</h4>
<p>Unless you are going to give the same presentation over and over again (and some do this, refining it over time), having your talk recorded means you instantly increase the size of your potential audience by 1000x. Not only does this force you to know what you are talking about and make pretty slides, but it helps justify the amount of work required to prepare.</p>
<p>Rather than having only static images to refer to or your sweet <em>(cough)</em> prezi, those wishing to review your presentation can do so directly from you in addition to the static version (see ‘Publishing’ below).</p>
<p>Some conferences will really go above and beyond in recording. Devoxx not only uses professional recording equipment with manned cameras, but they also publish their talks on <a href="https://parleys.com">parleys.com</a> which features slides that are synchronized with the video. You can <a href="http://parleys.com/play/529bde2ce4b0e619540cc3ae">watch my talk</a> from last year on Dagger for an example.</p>
<h4 id="venue">Venue</h4>
<p>With the amount of time it takes to prepare a presentation, you want it to be seen and heard by as many as possible. A bad venue can really turn an otherwise interesting set of speakers into a painful experience. Not being able to see or hear in a room is bad, but having to turn away tons of people due to poor planning or room size is even worse.</p>
<p>Beyond the physical, the technical facilities of a conference are of great importance. The size, placement, and resolution of the screen ensures that your content will be legible to all in the room. At this point it is inexcusable to have anything other than a widescreen, high-definition projector. Screen size should simply be as large as possible. For a presentation of anything over 20 or 30 a speaker and mic setup is essential. The size of the room should dictate how large the speaker setup needs to be.</p>
<p><img src="/static/post-image/devoxx_room.jpg" width="300" class="pull-right" /></p>
<p>This is another Devoxx slam-dunk as it is hosted inside a movie theater with stadium seating <em>(pictured)</em>. Presenters wear tiny mics and stand beneath an absolutely massive screen onto which both your slides and a real-time video feed is shown. As both an attendee and presenter it’s a fantastic experience.</p>
<h4 id="organizers">Organizers</h4>
<p>The people behind the conference is an interesting factor to guage a conference by but it turns out to be an important one. Some conferences are run by companies who have motives beyond the conference content itself. All conferences will have sponsors and may even be run by people who work for a single company. The difference is the companies for which running conferences is part of their job. By doing this, their motivation for recording talks, finding the best venues, and having the best equipment is greatly diminished by ulterior motives.</p>
<p>A conference that is run by a company is likely still a good conference to attend and will attract good speakers. It’s just a choice you have to make as to whether you want to support something like that. There is a well-known Android conference in San Francisco that I would love to see destroyed by a Droidcon SF since they are guilty of not doing the things listed above.</p>
<hr />
<h3 id="creating-your-slides">Creating Your Slides</h3>
<p>There are tons of articles on how to create effective slides. Slides should normally exist to augment and re-enforce you. Granted, some things like screenshots and code snippets are invaluable, and invert the roles allowing you to augment them. I’m not going re-iterate the well-known things and instead focus on what’s applicable for technical talks.</p>
<h4 id="getting-started">Getting Started</h4>
<p>There’s a variety of tools to choose from for creating slides, choose the one that best suits the job. Google Slides is fantastic for collaborating on content, Keynote has a fantastic balance of simplicity and power, and Javascript-based tools allow fantastic interactivity. There’s also PowerPoint, but I’m not a Windows user. All will work, choose the one which will make you the most productive and I’ll leave it to other articles to compare.</p>
<p>Slides should be done in widescreen (16:9) at 1080p (1920x1080) resolution. Hopefully this will be the native resolution at which they will be played (as discussed above), but at worst they will be downscaled which is always better than having them upscaled.</p>
<p>Never start a presentation with a description of who you are, what you do, and the outline of your slides. This is content for the abstract of your talk that most people will have read and used in choosing to attend. Those who didn’t read this information don’t need it because they chose your talk for its catchy title or something and thus won’t care either way.</p>
<h4 id="text--bullets">Text & Bullets</h4>
<p>Good ‘ol bullets of text. The fundamental building block of a presentation. Some people can get away with using only single words, short phrases, and pretty images but rarely does that fly for a technical talk.</p>
<p>There is no “too large” for font size of your text. Better to err on the side of large than too small. A single slide should only be able to fit 6 single-line bullets of text. What you think is readable when sitting two feet from your 23” monitor might not be readable at full scale. Start your slideshow and stand back 6 to 10 feet (depending on monitor size) and ensure that everything is readable.</p>
<p><img src="/static/post-image/slide_layout_comparison.gif" width="300" class="pull-right" style="border: 1px solid #eee;" /></p>
<p>Use simple layouts for text on slides which maximizes the available space and readability. The large size of your font tends to force this, but some slide layouts too heavily emphasize design rather than practicality. For example, Square has this one absolutely atrocious slide template <em>(pictured)</em> which put the title and bullets side-by-side and end up wasting 2/3rds of the screen real estate. It looks great when sitting in front of your laptop but in actual presentation only the first groupings of people will be able to read them with ease. The title should be at the top, bullets underneath, and font size at least doubled.</p>
<p>The text on the actual bullets is tricky to get right. You want enough information to convey the point you are trying to get across but not so much to force the audience spend most of their time reading. Sticking to a single line is usually a good litmus test for the right amount of content. Each bullet should animate in individually when you are talking about it. Having all the bullets on the screen at once will send most people off to read them all rather than listen to you.</p>
<h4 id="visualizations">Visualizations</h4>
<p>Conveying abstract or complicated concepts effectively with only text can be challenging. Visualizations can aid in helping the audience understand exactly what it is you are describing.</p>
<p>Similar to text, visualizations should be large and unambiguous. If there is more than 10 or 12 components you are going to saturate the individual’s ability to understand the content. Remember, while you are familiar with everything happening on screen this is the first time the attendees are seeing it.</p>
<p>Visualization should start simple and gradually enhance themselves if there are more than 5 or 6 components. In addition to bringing components in gradually, deliberate animation is also extremely helpful. The animation draws the eye to the correct place and should be used to help convey the concept which you are describing. In Keynote, break the visualization up into multiple slides and use the ‘Magic Move’ slide transition to handle figuring out most of the moves for you.</p>
<p>A visualization is not a substitute for a description of what is happening and why. Phrases like “as you can see” are red flags that you need to be the one explaining the behavior rather than relying on the visualization or animation to do it for you. Avoid trying to force the user where to look with commands. If there is an area you want to focus on, fade the unimportant parts of the visualization to 50% opacity to naturally draw their eyes to the important sections.</p>
<h4 id="code">Code</h4>
<p>What is a technical presentation without code? Showing code on screen is a very hard thing to do correctly. Code comes with an inherent cognitive overhead versus text, but it also is fraught with opinion in both language and style.</p>
<p>It should come as no surprise that yet again size is important here. You don’t need to go as large as text, but limit the number of lines of code to 12 to 15 on a slide. Similar to the gradual enhancement in visualizations, if you need to show more than 5 or 6 lines of code break it up into logical chunks and fade them in individually. Explain each section as it comes in and then summarize in one line again once the complete snippet is on the screen to reinforce the behavior.</p>
<p>Syntax highlighting of code is essential for slides. Programmer’s eyes are trained to recognize the semantic building blocks of code through syntax highlighting. There is absolutely no reason to omit this. IntelliJ 14 or newer will do rich-text copy with syntax highlighting and can even be customized to use a different style when copying than the active one (search ‘rich’ in preferences).</p>
<p><img src="/static/post-image/slide_code_diff.gif" width="300" class="pull-right" /></p>
<p>Often when showing code in a presentation, you will want to show code snippets that change between slides as a result of demonstrating new APIs or just as a different example. Back-to-back slides with syntax-highlighted code is hard on your audience because they need to both understand what changed and then understand why. Every single change in code you make should initially desaturate the unchanged parts and then automatically re-saturate after a short interval <em>(pictured)</em>. Duplicate the original slide and update it with the changed code. Copy and paste the code on the new slide and position it at the same <em>x</em> and <em>y</em> positions. Select the parts of the code which haven’t changed and set them to a neutral gray. Put a fade-out animation on the desaturated code of 0.5 seconds with a delay of 1.5 seconds and set it to play automatically when the slide is shown. Put a 0.5 second cross-fade transition between the two slides.</p>
<p><img src="/static/post-image/slide_code_highlight.gif" width="300" class="pull-right" /></p>
<p>The same desaturation technique should be used whenever you need to emphasize a section of code that is already on screen. Here, however, it’s wise to do it as explicit animation steps for both desaturation and re-saturation since you want the focus to be kept for as long as you are talking about that code <em>(pictured)</em>. Use the same technique as above with separate slides with the exception of having the desaturated code fade out on click.</p>
<h4 id="publishing">Publishing</h4>
<p>Unless a conference has really professional recording and post-processing the quality of your slides on the video will be low. Publishing them separately allows for a much higher quality and gives control to the viewer as to how quickly or slowly they want to move forward. If you have followed the above recommendations your slides can mostly likely stand on their own without the video. Those referring back to them won’t need to use the recording and those not in attendance can look at them while waiting for the video to be released.</p>
<p><a href="https://speakerdeck.com/jakewharton/">SpeakerDeck</a> is my preferred publishing platform of choice. It has a no-frills interface without ads or noisy chrome around the slides like other sites. It also allows embedding if you have your own website that chronicles your presentations.</p>
<p>To export your slides, make a copy of the original version and append ‘-ForExport’. You will almost certainly have to make changes for export and operating only on a copy ensures you don’t accidentally overwrite the original. After opening the copy, immediately export a PDF. Be sure to select the options that creates a separate slide for each animation step and include the slide number. Open the PDF and flip through from start to end noting any changes you want to make in a text editor with the slide number. Flip back to the presentation and make the changes, re-export, and repeat until you have something you are satisfied with. The two most common changes I have to make is removing animations which play immediately after transition and splitting slides with complex move animations into multiple.</p>
<p>Once you are satisfied with your PDF, do one final export which disables the slide numbers and sets the image quality to its highest setting. Save both the ‘ForExport’ version of the presentation and the final exported PDF along side the original. I’ve had to circle back and tweak the export or email the exported PDF on multiple occasions. Upload the presentation to SpeakerDeck, give it a good title and description, and check the little publish box. Once the video is made available, be sure to update the description with a link to the video.</p>
<hr />
<p>These are my opinions having done this only about 10 times. By no means am I an expert, but I think I have good grasp on how to choose the presentations that I give. And if you didn’t pick up on it, Devoxx and Keynote are my current bar by which I measure the venue and presentation quality, respectively.</p>
<p>I’ve uploaded some files from my most recent presentation on Dagger 2 as examples:</p>
<ul>
<li>Original Keynote (zip): <a href="https://drive.google.com/open?id=0B490UMAh3G13TjBrTGQ2OVBFdjA">drive.google.com/open?id=0B490UMAh3G13TjBrTGQ2OVBFdjA</a></li>
<li>Original PDF Export: <a href="https://drive.google.com/open?id=0B490UMAh3G13YjZUNnNoaHluWmc">drive.google.com/open?id=0B490UMAh3G13YjZUNnNoaHluWmc</a></li>
<li>‘ForExport’ Keynote (zip): <a href="https://drive.google.com/open?id=0B490UMAh3G13U3pOZ0o0WXNUXzA">drive.google.com/open?id=0B490UMAh3G13U3pOZ0o0WXNUXzA</a></li>
<li>Final PDF Export: <a href="https://drive.google.com/open?id=0B490UMAh3G13ZlFfVVdiZDJvcGs">drive.google.com/open?id=0B490UMAh3G13ZlFfVVdiZDJvcGs</a></li>
<li>SpeakerDeck upload: <a href="https://speakerdeck.com/jakewharton/dependency-injection-with-dagger-2-devoxx-2014">speakerdeck.com/jakewharton/dependency-injection-with-dagger-2-devoxx-2014</a></li>
</ul>
Coercing Picasso To Play With Palette2014-10-24T00:00:00+00:00https://jakewharton.com/coercing-picasso-to-play-with-palette<p>Features feel how I imagine children would. They are relatively easy to spawn but require a lengthy committment and constant care. Picasso’s API would be nothing short of a Greek tragedy if we humored every feature request that we received. Finding the right balance of what is appropriate to add and what isn’t is a constant struggle.</p>
<p>When <a href="https://developer.android.com/tools/support-library/features.html#v7-palette">the Palette library</a> was teased the inevitable feature request came in to Picasso for a means of supporting it. This was certainly not an unreasonable request, and while we might not explicitly support it directly we will probably add something in the future to more easily facilitate its use.</p>
<p>But what do we do in the interim? The API for proper support is many months if not a year away from actually being implemented. Let’s walk through an attempt to adapt the existing APIs to allow Palette’s use.</p>
<hr />
<p>The fundamental component of Picasso’s data pipeline after the request has been fulfilled is a <code class="highlighter-rouge">Bitmap</code>. We use this in the return values and method parameters which traverse upwards to the main thread. <code class="highlighter-rouge">Bitmap</code> is a <code class="highlighter-rouge">final</code> class in Android so we will not get an opportunity to hang extra metadata on a subclass.</p>
<p>There are two ways that Picasso can notify the caller of a successful image download: the <code class="highlighter-rouge">Callback</code> when loading directly into an <code class="highlighter-rouge">ImageView</code> or the more generalized <code class="highlighter-rouge">Target</code>. A <code class="highlighter-rouge">Target</code> is given direct access to the <code class="highlighter-rouge">Bitmap</code> object but the <code class="highlighter-rouge">Callback</code> is not (although you can get it indirectly with some cleverness). Regardless of <code class="highlighter-rouge">Bitmap</code> access is the problem that both of these are called on the main thread. Since Palette does a decent amount of computation we don’t want to do it here anyways.</p>
<p>Palette actually has an asynchronous mode of operation that we could leverage in these two callback locations but you wouldn’t want to. The images are ready to be displayed when the callbacks are invoked so either delaying display until you can run Palette on another background thread or displaying the image right away and getting the Palette information later both seem like sub-par experiences.</p>
<p>From the time Picasso gets the <code class="highlighter-rouge">Bitmap</code> to the time it makes it back to the main thread, where can we hook in to allow the invocation of Palette inside of the threading model which is already being managed by Picasso? Those familiar with Picasso will know that there’s two places: the <code class="highlighter-rouge">Downloader</code> (or <code class="highlighter-rouge">RequestHandler</code> in the upcoming v2.4) and in a <code class="highlighter-rouge">Transformer</code>.</p>
<p><code class="highlighter-rouge">Downloader</code> and the upcoming <code class="highlighter-rouge">RequestHandler</code> are means to obtaining the original <code class="highlighter-rouge">Bitmap</code> instances (or <code class="highlighter-rouge">InputStream</code> instances) to fulfill a request. While we could invoke Palette here, I’m going to immediately reject it for a few reasons. There are multiple sources from which an image can be loaded which means we need to duplicate our logic across all of them. Additionally, the number of sources is constantly changing and some of them you cannot replace (I’m looking at you, drawable resource ID loading). Sometimes sources provide an <code class="highlighter-rouge">InputStream</code> which means we now have the burden of doing the initial <code class="highlighter-rouge">Bitmap</code> decoding ourselves–a job which is supposed to be Picasso’s, right? Finally, the <code class="highlighter-rouge">Bitmap</code> at this level is the raw, original sized version. Not only will Palette operate much more slowly on it but a later transformation might alter the color makeup of the image.</p>
<p>A <code class="highlighter-rouge">Transformer</code>, it would seem, remains our only hope. And in fact, the more you look at it the more appealing it becomes. A <code class="highlighter-rouge">Transformer</code> always receives a <code class="highlighter-rouge">Bitmap</code> instance. It is invoked after all of the internal transformations have been applied. This means that the ever-common <code class="highlighter-rouge">fit()</code>/<code class="highlighter-rouge">resize()</code> & <code class="highlighter-rouge">centerCrop()</code> combo have already been executed. All transformers run last in the pipeline and the order is controlled by the caller. This means that we can place a custom transformer as the very last thing that is run before Picasso starts the process of sending the <code class="highlighter-rouge">Bitmap</code> back to the main thread.</p>
<p>I think we found our hook. Let’s get started on some code:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">final</span> <span class="nc">PaletteTransformation</span> <span class="kd">implements</span> <span class="nc">Transformation</span> <span class="o">{</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="nc">Bitmap</span> <span class="nf">transform</span><span class="o">(</span><span class="nc">Bitmap</span> <span class="n">source</span><span class="o">)</span> <span class="o">{</span>
<span class="c1">// TODO Palette all the things!</span>
<span class="k">return</span> <span class="n">source</span><span class="o">;</span>
<span class="o">}</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="nc">String</span> <span class="nf">key</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="s">""</span><span class="o">;</span> <span class="c1">// Stable key for all requests. An unfortunate requirement.</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<hr />
<p>While <code class="highlighter-rouge">Transformer</code> gives us the very last minute hook into the processing pipeline, as we noted before, its return type of <code class="highlighter-rouge">Bitmap</code> means we aren’t hanging any metadata directly on the return value. How can we propogate this metadata between the transformation back to the call site?</p>
<p>Looking at how we invoke Picasso with our transformation should give you a clue:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">Picasso</span><span class="o">.</span><span class="na">with</span><span class="o">(</span><span class="n">context</span><span class="o">)</span>
<span class="o">.</span><span class="na">load</span><span class="o">(</span><span class="n">url</span><span class="o">)</span>
<span class="o">.</span><span class="na">fit</span><span class="o">().</span><span class="na">centerCrop</span><span class="o">()</span>
<span class="o">.</span><span class="na">transform</span><span class="o">(</span><span class="k">new</span> <span class="nc">PaletteTransformation</span><span class="o">())</span>
<span class="o">.</span><span class="na">into</span><span class="o">(</span><span class="n">imageView</span><span class="o">,</span> <span class="k">new</span> <span class="nc">EmptyCallback</span><span class="o">()</span> <span class="o">{</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">onSuccess</span><span class="o">()</span> <span class="o">{</span>
<span class="c1">// TODO I can haz Palette?</span>
<span class="o">}</span>
<span class="o">});</span>
</code></pre></div></div>
<p>Those familiar with Picasso best practices should be screaming about the <code class="highlighter-rouge">new PaletteTransformer()</code> snippet. In general, all Picasso transformations should be completely stateless functions so that a single instance can be used for every call to <code class="highlighter-rouge">.transform()</code>. In this case we are going to make an exception because the transformation looks like a great place to pass along our metadata.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">final</span> <span class="nc">PaletteTransformation</span> <span class="n">paletteTransformation</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">PaletteTransformation</span><span class="o">();</span>
<span class="nc">Picasso</span><span class="o">.</span><span class="na">with</span><span class="o">(</span><span class="n">context</span><span class="o">)</span>
<span class="o">.</span><span class="na">load</span><span class="o">(</span><span class="n">url</span><span class="o">)</span>
<span class="o">.</span><span class="na">fit</span><span class="o">().</span><span class="na">centerCrop</span><span class="o">()</span>
<span class="o">.</span><span class="na">transform</span><span class="o">(</span><span class="n">paletteTransformation</span><span class="o">)</span>
<span class="o">.</span><span class="na">into</span><span class="o">(</span><span class="n">imageView</span><span class="o">,</span> <span class="k">new</span> <span class="nc">EmptyCallback</span><span class="o">()</span> <span class="o">{</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">onSuccess</span><span class="o">()</span> <span class="o">{</span>
<span class="nc">Palette</span> <span class="n">palette</span> <span class="o">=</span> <span class="n">paletteTransformation</span><span class="o">.</span><span class="na">getPalette</span><span class="o">();</span>
<span class="c1">// TODO apply palette to text views, backgrounds, etc.</span>
<span class="o">}</span>
<span class="o">});</span>
</code></pre></div></div>
<p>Now that we have a working model to hand off the metadata, let’s update our <code class="highlighter-rouge">PaletteTransformation</code> to actually extract the palette from the <code class="highlighter-rouge">Bitmap</code> that is passing through.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">final</span> <span class="nc">PaletteTransformation</span> <span class="kd">implements</span> <span class="nc">Transformation</span> <span class="o">{</span>
<span class="kd">private</span> <span class="nc">Palette</span> <span class="n">palette</span><span class="o">;</span>
<span class="kd">public</span> <span class="nc">Palette</span> <span class="nf">getPalette</span><span class="o">()</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span><span class="n">palette</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">IllegalStateException</span><span class="o">(</span><span class="s">"Transformation was not run."</span><span class="o">);</span>
<span class="o">}</span>
<span class="k">return</span> <span class="n">palette</span><span class="o">;</span>
<span class="o">}</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="nc">Bitmap</span> <span class="nf">transform</span><span class="o">(</span><span class="nc">Bitmap</span> <span class="n">source</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span><span class="n">palette</span> <span class="o">!=</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">IllegalStateException</span><span class="o">(</span><span class="s">"Instances may only be used once."</span><span class="o">);</span>
<span class="o">}</span>
<span class="n">palette</span> <span class="o">=</span> <span class="nc">Palette</span><span class="o">.</span><span class="na">generate</span><span class="o">(</span><span class="n">source</span><span class="o">);</span>
<span class="k">return</span> <span class="n">source</span><span class="o">;</span>
<span class="o">}</span>
<span class="c1">// ...</span>
<span class="o">}</span>
</code></pre></div></div>
<p>While this looks like a working solution, there are two problems:</p>
<ol>
<li>It requires an additional object allocation for every Picasso request (something we try very hard at to minimize).</li>
<li>Images which are cached do not pass through the transformation pipeline and thus will always return <code class="highlighter-rouge">null</code> from <code class="highlighter-rouge">getPalette()</code>.</li>
</ol>
<hr />
<p>Object allocations are becoming more cheap with newer platform versions but will never be free. Since Picasso is often called thousands of times in very performance sensitive areas of applications we aim for the utmost efficiency in terms of CPU and memory use.</p>
<p>Saving the allocation for our transformation is easy. We can use the well-known pattern of object pooling. Thanks to recent support library updates, this is even easier to do than before with <a href="https://developer.android.com/reference/android/support/v4/util/Pools.html">the <code class="highlighter-rouge">Pools</code> helper</a>.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">final</span> <span class="nc">PaletteTransformation</span> <span class="kd">implements</span> <span class="nc">Transformation</span> <span class="o">{</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">Pool</span><span class="o"><</span><span class="nc">PaletteTransformation</span><span class="o">></span> <span class="no">POOL</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">SynchronizedPool</span><span class="o"><>(</span><span class="mi">5</span><span class="o">);</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="nc">PaletteTransformation</span> <span class="nf">getInstance</span><span class="o">()</span> <span class="o">{</span>
<span class="nc">PaletteTransformation</span> <span class="n">instance</span> <span class="o">=</span> <span class="no">POOL</span><span class="o">.</span><span class="na">obtain</span><span class="o">();</span>
<span class="k">return</span> <span class="n">instance</span> <span class="o">!=</span> <span class="kc">null</span> <span class="o">?</span> <span class="n">instance</span> <span class="o">:</span> <span class="k">new</span> <span class="nc">PaletteTransformation</span><span class="o">();</span>
<span class="o">}</span>
<span class="kd">private</span> <span class="nc">Palette</span> <span class="n">palette</span><span class="o">;</span>
<span class="kd">private</span> <span class="nf">PaletteTransformation</span><span class="o">()</span> <span class="o">{}</span>
<span class="kd">public</span> <span class="nc">Palette</span> <span class="nf">extractPaletteAndRelease</span><span class="o">()</span> <span class="o">{</span>
<span class="nc">Palette</span> <span class="n">palette</span> <span class="o">=</span> <span class="k">this</span><span class="o">.</span><span class="na">palette</span><span class="o">;</span>
<span class="k">if</span> <span class="o">(</span><span class="n">palette</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">IllegalStateException</span><span class="o">(</span><span class="s">"Transformation was not run."</span><span class="o">);</span>
<span class="o">}</span>
<span class="k">this</span><span class="o">.</span><span class="na">palette</span> <span class="o">=</span> <span class="kc">null</span><span class="o">;</span>
<span class="no">POOL</span><span class="o">.</span><span class="na">release</span><span class="o">(</span><span class="k">this</span><span class="o">);</span>
<span class="k">return</span> <span class="n">palette</span><span class="o">;</span>
<span class="o">}</span>
<span class="c1">// ...</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Our calling code only changes slightly to use the new static factory and the more semantically named palette extraction method.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">final</span> <span class="nc">PaletteTransformation</span> <span class="n">paletteTransformation</span> <span class="o">=</span> <span class="nc">PaletteTransformation</span><span class="o">.</span><span class="na">getInstance</span><span class="o">();</span>
<span class="nc">Picasso</span><span class="o">.</span><span class="na">with</span><span class="o">(</span><span class="n">context</span><span class="o">)</span>
<span class="o">.</span><span class="na">load</span><span class="o">(</span><span class="n">url</span><span class="o">)</span>
<span class="o">.</span><span class="na">fit</span><span class="o">().</span><span class="na">centerCrop</span><span class="o">()</span>
<span class="o">.</span><span class="na">transform</span><span class="o">(</span><span class="n">paletteTransformation</span><span class="o">)</span>
<span class="o">.</span><span class="na">into</span><span class="o">(</span><span class="n">imageView</span><span class="o">,</span> <span class="k">new</span> <span class="nc">EmptyCallback</span><span class="o">()</span> <span class="o">{</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">onSuccess</span><span class="o">()</span> <span class="o">{</span>
<span class="nc">Palette</span> <span class="n">palette</span> <span class="o">=</span> <span class="n">paletteTransformation</span><span class="o">.</span><span class="na">extractPaletteAndRelease</span><span class="o">();</span>
<span class="c1">// TODO apply palette to text views, backgrounds, etc.</span>
<span class="o">}</span>
<span class="o">});</span>
</code></pre></div></div>
<p>I have hard coded the pool size to retain 5 instances. This is an educated guess based on how I know Picasso’s internals to work. If you were adopting this implementation you should add logging to test whether the size needs to be increased for you application.</p>
<hr />
<p>Dealing with memory-cached images is a bit less straightforward. Picasso’s <code class="highlighter-rouge">Cache</code> is hard coded to use <code class="highlighter-rouge">Bitmap</code> as the value which means we can’t wrap up the <code class="highlighter-rouge">Palette</code> instance along side. Since we can’t use the main cache we will be forced to mirror it.</p>
<p>In deciding on our cache key we have a choice: the <code class="highlighter-rouge">Bitmap</code> which is the source of truth for the pixels or the URL which is the source of truth for the image. The choice here leads to two different implementations and neither one is applicable to all use cases. We’ll quickly explore both which will yield the final result.</p>
<p>Keying by <code class="highlighter-rouge">Bitmap</code> creates the most elegant implementation of the transformation at the expense of ugliness in the calling code. We can even revert to using a single transformation instance with an embedded cache. Pooling is fun but not having to pool is even better!</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">final</span> <span class="kd">class</span> <span class="nc">PaletteTransformation</span> <span class="kd">implements</span> <span class="nc">Transformation</span> <span class="o">{</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">PaletteTransformation</span> <span class="no">INSTANCE</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">PaletteTransformation</span><span class="o">();</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">Map</span><span class="o"><</span><span class="nc">Bitmap</span><span class="o">,</span> <span class="nc">Palette</span><span class="o">></span> <span class="no">CACHE</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">WeakHashMap</span><span class="o"><>();</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="nc">PaletteTransformation</span> <span class="nf">instance</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="no">INSTANCE</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="nc">Palette</span> <span class="nf">getPalette</span><span class="o">(</span><span class="nc">Bitmap</span> <span class="n">bitmap</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="no">CACHE</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">bitmap</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">private</span> <span class="nf">PaletteTransformation</span><span class="o">()</span> <span class="o">{}</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="nc">Bitmap</span> <span class="nf">transform</span><span class="o">(</span><span class="nc">Bitmap</span> <span class="n">source</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">Palette</span> <span class="n">palette</span> <span class="o">=</span> <span class="nc">Palette</span><span class="o">.</span><span class="na">generate</span><span class="o">(</span><span class="n">source</span><span class="o">);</span>
<span class="no">CACHE</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="n">source</span><span class="o">,</span> <span class="n">palette</span><span class="o">);</span>
<span class="k">return</span> <span class="n">source</span><span class="o">;</span>
<span class="o">}</span>
<span class="c1">// ...</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The <code class="highlighter-rouge">WeakHashMap</code> will release the <code class="highlighter-rouge">Palette</code> reference when its associated <code class="highlighter-rouge">Bitmap</code> is garbage collected. We rely on Picasso’s memory cache to retain the strong reference even if it isn’t currently being displayed in an <code class="highlighter-rouge">ImageView</code>.</p>
<p>The calling code has to obtain the final <code class="highlighter-rouge">Bitmap</code> in order to query the cache. This is trivial in Picasso’s <code class="highlighter-rouge">Target</code>, but much more ugly in the more common <code class="highlighter-rouge">Callback</code>.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">Picasso</span><span class="o">.</span><span class="na">with</span><span class="o">(</span><span class="n">context</span><span class="o">)</span>
<span class="o">.</span><span class="na">load</span><span class="o">(</span><span class="n">url</span><span class="o">)</span>
<span class="o">.</span><span class="na">fit</span><span class="o">().</span><span class="na">centerCrop</span><span class="o">()</span>
<span class="o">.</span><span class="na">transform</span><span class="o">(</span><span class="nc">PaletteTransformation</span><span class="o">.</span><span class="na">instance</span><span class="o">())</span>
<span class="o">.</span><span class="na">into</span><span class="o">(</span><span class="n">imageView</span><span class="o">,</span> <span class="k">new</span> <span class="nc">EmptyCallback</span><span class="o">()</span> <span class="o">{</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">onSuccess</span><span class="o">()</span> <span class="o">{</span>
<span class="nc">Bitmap</span> <span class="n">bitmap</span> <span class="o">=</span> <span class="o">((</span><span class="nc">BitmapDrawable</span><span class="o">)</span> <span class="n">imageView</span><span class="o">.</span><span class="na">getDrawable</span><span class="o">()).</span><span class="na">getBitmap</span><span class="o">();</span> <span class="c1">// Ew!</span>
<span class="nc">Palette</span> <span class="n">palette</span> <span class="o">=</span> <span class="nc">PaletteTransformation</span><span class="o">.</span><span class="na">getPalette</span><span class="o">(</span><span class="n">bitmap</span><span class="o">);</span>
<span class="c1">// TODO apply palette to text views, backgrounds, etc.</span>
<span class="o">}</span>
<span class="o">});</span>
</code></pre></div></div>
<p>This approach places each <code class="highlighter-rouge">Bitmap</code> as the source of truth for the <code class="highlighter-rouge">Palette</code> instance. The same URL which is displayed at different resolutions might have slightly different values in each swatch because of this. Using the URL as the key on a cache would ensure that multiple sizes of the same image had exactly the same palette. I’m not exactly sure of how much of an issue this is in practice, if any.</p>
<p>There are other problems with a URL <code class="highlighter-rouge">String</code> key approach, however. Picasso’s default <code class="highlighter-rouge">LruCache</code> implementation does not expose a callback for when entries are being purged. This means we have no way of reference counting the <code class="highlighter-rouge">Palette</code> instances in a parallel cache (remember, multiple entries in the main cache could reference the same <code class="highlighter-rouge">Palette</code>). We could create a new <code class="highlighter-rouge">Cache</code> implementation based on the support-v4 library <code class="highlighter-rouge">LruCache</code> (from which Picasso’s is based) but now we are duplicating functionality for little gain.</p>
<p>Another way to solve the problem would be a double-key map where the URL <code class="highlighter-rouge">String</code> is mapped to the <code class="highlighter-rouge">Palette</code> instance, but also each resulting transformed <code class="highlighter-rouge">Bitmap</code> instance was used as a map key with a weak reference. When a reference queue callback was invoked because the <code class="highlighter-rouge">Bitmap</code> was garbage collected, we check to see whether any <code class="highlighter-rouge">Bitmap</code> mappings still exist and if not purge the <code class="highlighter-rouge">String</code> to <code class="highlighter-rouge">Palette</code> mapping. This is viable, but it’s more work than I am willing to do because ultimately we are designing a temporary solution.</p>
<p>It looks like the <code class="highlighter-rouge">Bitmap</code> key based approach is the most viable at the expense of potential (but not proven) variance in the <code class="highlighter-rouge">Palette</code> instances for multiple <code class="highlighter-rouge">Bitmaps</code> of the same image URL.</p>
<hr />
<p>This is a long post and there isn’t exactly a nice neat bow to tie it all together with. I wrote it in this way because I wanted to showcase two very important things when it comes to how we do feature and API design:</p>
<ol>
<li>Not every use case deserves a first-party API but with some clever thinking you can usually accomplish what you are after by thinking outside of the box. We ultimately will support the use of Palette in Picasso but less as a first-class citizen and more through a general means of passing along arbitrary metadata through the pipeline.</li>
<li>Exhausting multiple implementations and solutions to a problem is essential for ironing out the best approach. This was a journey to the final conclusion and it’s one we frequently make for most of the changes we make to Picasso and all of our open source libraries. Not only can we be sure that we reached a good solution, but we also are able to defend and justify our decisions.</li>
</ol>
<p>If you come upon any other bright ideas for applying Palette into Picasso I’d love to hear about them. Otherwise get playing with the fantastic Palette library today and keep an eye out for Picasso 2.4 in the next few days!</p>
Play Services 5.0 Is A Monolith Abomination2014-07-03T00:00:00+00:00https://jakewharton.com/play-services-is-a-monolith<p><a href="https://code.google.com/p/guava-libraries/">Guava</a> is a monolithic library, but that’s not necessarily a bad thing. Nobody thinks twice when bundling it for the JVM. In the world of Android the mention of Guava has a bit of a negative stigma due to the dex file format’s method limit and a concern about bloating APK size. The latter is no longer a valid argument. The dex method limit is a hard 64k limit to which Guava contributes just over 14k methods. 20% of this hard limit vanishes when you include Guava.</p>
<p>Sounds scary, right? It isn’t.</p>
<p>Google Play Services 5.0 which <a href="http://android-developers.blogspot.com/2014/07/google-play-services-5.html">just launched</a> contributes over <strong>twenty thousand</strong> methods to your app. 20k+. One third of the limit! Now <em>that</em> is scary.</p>
<p>The Play Services library includes proprietary functionality built on the normal Android APIs and a separate APK downloaded on all devices with the Play Store. Some of the services it provides are invaluable. Like Guava it is also a monolothic library but it <em>is</em> a bad thing in this case.</p>
<p>A lot of really cool functionality is being put in Play Services. You’ll have a hard time making a compelling app that lives in the Google Play ecosystem without it. You should want to put it in your applications and not have to worry about the overhead it brings.</p>
<p>Most of the library’s offerings are very disparate, having only the fact that they’re by Google as a common thread. This screams for small, modular artifacts which can be composed!</p>
<p><strong>Google, it’s time to unbundle.</strong> <a href="http://thenextweb.com/socialmedia/2014/05/06/large-tech-companies-hopping-app-unbundling-trend/">All the cool kids are doing it</a>. <em>(Spoiler alert: <a href="http://android-developers.blogspot.com/2014/12/google-play-services-and-dex-method.html">it happened</a>)</em></p>
<p>At worst, we specify a few dependencies manually:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">dependencies</span> <span class="o">{</span>
<span class="n">compile</span> <span class="s1">'com.google.android.gms:play-services-ads:5.0.+'</span>
<span class="n">compile</span> <span class="s1">'com.google.android.gms:play-services-analytics:5.0.+'</span>
<span class="n">compile</span> <span class="s1">'com.google.android.gms:play-services-games:5.0.+'</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Best case would be a plugin that provided a clear DSL to what you were getting and offered easier configuration of the various components.</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">apply</span> <span class="nl">plugin:</span> <span class="s1">'com.google.playservices'</span>
<span class="n">playServices</span> <span class="o">{</span>
<span class="n">version</span> <span class="s1">'5.0.+'</span>
<span class="n">components</span> <span class="s1">'ads'</span><span class="o">,</span> <span class="s1">'analytics'</span><span class="o">,</span> <span class="s1">'games'</span>
<span class="o">}</span>
</code></pre></div></div>
<p>(You can even still provide the “fat” jar in both the dependency management world and the people who like manual dependency management.)</p>
<p>ProGuard is not the answer. Yes, for release builds it’s nice to strip out any methods which are not being used. However, this is not justification for having large chunks of unused code as dependencies. Besides, if you read <a href="http://jakewharton.com/android-needs-a-simulator/">my post on a simulator</a> you know that we deserve a faster development build pipeline which removes steps, not adds them.</p>
<p>It’s not going to be a walk in the park but the packages inside Play Services are surprisingly well-configured to partitioning:</p>
<p><a href="/static/post-image/play-services-deps.png"><img src="/static/post-image/play-services-deps.png" alt="Play Services Dependency Diagram" /></a></p>
<p><em>(Top-left: Games, top-center: Drive, middle-left: Plus, middle: common, middle-right: Maps, bottom: Ads)</em></p>
<p>Here’s Guava for comparison which has less clear partition lines:</p>
<p><a href="/static/post-image/guava-deps.png"><img src="/static/post-image/guava-deps.png" alt="Guava Dependency Diagram" /></a></p>
<hr />
<p>Here’s how the method counts were determined:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ curl 'http://search.maven.org/remotecontent?filepath=com/google/guava/guava/17.0/guava-17.0.jar' > guava.jar
$ ~/android-sdk/build-tools/20.0.0/dx --dex --output guava.dex guava.jar
$ dex-method-count guava.dex
14824
$ cp ~/android-sdk/extras/google/m2repository/com/google/android/gms/play-services/5.0.77/play-services-5.0.77.aar .
$ unzip play-services-5.0.77.aar
$ ~/android-sdk/build-tools/20.0.0/dx --dex --output play-services.dex classes.jar
$ dex-method-count play-services.dex
20298
</code></pre></div></div>
<p>And the full by-package breakdown of Play Services:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ dex-method-count-by-package play-services.dex
20298 com
20298 com.google
207 com.google.ads
169 com.google.ads.mediation
73 com.google.ads.mediation.admob
62 com.google.ads.mediation.customevent
20188 com.google.android
20188 com.google.android.gms
2 com.google.android.gms.actions
480 com.google.android.gms.ads
135 com.google.android.gms.ads.doubleclick
25 com.google.android.gms.ads.identifier
88 com.google.android.gms.ads.mediation
4 com.google.android.gms.ads.mediation.admob
73 com.google.android.gms.ads.mediation.customevent
26 com.google.android.gms.ads.purchase
118 com.google.android.gms.ads.search
866 com.google.android.gms.analytics
52 com.google.android.gms.analytics.ecommerce
10 com.google.android.gms.appindexing
151 com.google.android.gms.appstate
80 com.google.android.gms.auth
644 com.google.android.gms.cast
1026 com.google.android.gms.common
12 com.google.android.gms.common.annotation
382 com.google.android.gms.common.api
235 com.google.android.gms.common.data
202 com.google.android.gms.common.images
126 com.google.android.gms.common.internal
126 com.google.android.gms.common.internal.safeparcel
1940 com.google.android.gms.drive
87 com.google.android.gms.drive.events
897 com.google.android.gms.drive.internal
241 com.google.android.gms.drive.metadata
202 com.google.android.gms.drive.metadata.internal
205 com.google.android.gms.drive.query
151 com.google.android.gms.drive.query.internal
451 com.google.android.gms.drive.realtime
451 com.google.android.gms.drive.realtime.internal
123 com.google.android.gms.drive.realtime.internal.event
38 com.google.android.gms.drive.widget
332 com.google.android.gms.dynamic
4534 com.google.android.gms.games
73 com.google.android.gms.games.achievement
113 com.google.android.gms.games.event
2956 com.google.android.gms.games.internal
858 com.google.android.gms.games.internal.api
43 com.google.android.gms.games.internal.constants
8 com.google.android.gms.games.internal.data
31 com.google.android.gms.games.internal.events
9 com.google.android.gms.games.internal.experience
215 com.google.android.gms.games.internal.game
56 com.google.android.gms.games.internal.multiplayer
23 com.google.android.gms.games.internal.notification
80 com.google.android.gms.games.internal.player
86 com.google.android.gms.games.internal.request
256 com.google.android.gms.games.leaderboard
640 com.google.android.gms.games.multiplayer
239 com.google.android.gms.games.multiplayer.realtime
256 com.google.android.gms.games.multiplayer.turnbased
213 com.google.android.gms.games.quest
150 com.google.android.gms.games.request
210 com.google.android.gms.games.snapshot
47 com.google.android.gms.gcm
111 com.google.android.gms.identity
111 com.google.android.gms.identity.intents
62 com.google.android.gms.identity.intents.model
5760 com.google.android.gms.internal
295 com.google.android.gms.location
2342 com.google.android.gms.maps
804 com.google.android.gms.maps.internal
1068 com.google.android.gms.maps.model
483 com.google.android.gms.maps.model.internal
14 com.google.android.gms.panorama
902 com.google.android.gms.plus
352 com.google.android.gms.plus.internal
316 com.google.android.gms.plus.model
192 com.google.android.gms.plus.model.moments
126 com.google.android.gms.plus.model.people
33 com.google.android.gms.security
1367 com.google.android.gms.tagmanager
867 com.google.android.gms.wallet
376 com.google.android.gms.wallet.fragment
143 com.google.android.gms.wallet.wobs
1011 com.google.android.gms.wearable
714 com.google.android.gms.wearable.internal
</code></pre></div></div>
<p>You can grab these two scripts from here: <a href="https://gist.github.com/JakeWharton/6002797">gist.github.com/JakeWharton/6002797</a></p>
<p>The dependency graphs were generated using <a href="https://github.com/schauder/degraph">degraph</a> and <a href="http://www.yworks.com/en/products_yed_about.html">yEd</a>. Download the <code class="highlighter-rouge">.graphml</code> for <a href="/static/files/play-services-5.graphml">Play Services</a> and <a href="/static/files/guava-17.graphml">Guava</a>.</p>
Android Needs A Simulator, Not An Emulator2014-06-16T00:00:00+00:00https://jakewharton.com/android-needs-a-simulator<p>Two years ago I wrote <a href="http://jakewharton.com/the-android-build-system-is-broken/">a blog post</a> complaining that the Android build system was broken. At the time, Eclipse ADT and Ant were the blessed solutions and they just hadn’t scaled with the platform. Third-party solutions existed for both tooling and IDE but they always felt a bit illegitimate and at risk for problems. My post joined the cries of others who knew that something <em>had</em> to be done.</p>
<p><a href="https://google.com/+XavierDucrohet">Xavier Ducrohet</a> swooped in and dropped a bomb on the <a href="https://plus.google.com/+JakeWharton/posts/KuWYBtLKtSE">resulting Google+ thread</a>: “We are looking at revamping the whole thing”.</p>
<p>In the two years since he and the tools team have transformed the landscape of how Android development is done. A first-party Gradle plugin now provides the powerful and dynamic platform on which any app of quality is built. Ownership of the Android plugin inside IntelliJ IDEA (with a sprinkle of branding) yields a development environment that moves mountains for you.</p>
<p>Neither the Gradle plugin nor the IntelliJ IDEA plugin (known bundled as Android Studio) are at a v1.0 yet. They’re both still beta (albeit arguably in the sense that GMail was circa 2008).</p>
<p>Why was all of this important and why is it important for Google moving forward?</p>
<p>Developers are the top of the funnel for Android’s continued success. Without quality development tools there are no quality apps, without quality apps there are no quality users, and without quality users the developers will flee. Half of the developer flow in this funnel comes from the tools and the other half from APIs. This post is about the former.</p>
<hr />
<p><img src="/static/post-image/emulator-m3.png" width="300" class="pull-right" /></p>
<p>My first exposure to Android was the M3 pre-release SDK’s emulator <em>(pictured)</em>. The emulator started up quickly and was responsive. Each successive release up to and beyond version 1.0 added much needed functionality to both the OS and the emulator. And in each successive release the emulator slowed.</p>
<p>Android’s pubescent period (otherwise known as Honeycomb) and its eventual emergence into adulthood had a devestating effect on emulator performance. Despite our development environments becoming more powerful, two factors outpaced Moore’s law:</p>
<ol>
<li>Support of tablets and advances in screen technology meant that devices were gaining a lot of pixels.</li>
<li>A more advanced graphics pipeline pushed rendering from software down into the hardware. This brought great performance at the expense of internal complexity.</li>
</ol>
<p>Working with Intel, Google eventually <a href="http://android-developers.blogspot.com/2012/04/faster-emulator-with-better-hardware.html">released an x86 version</a> of the emulator which eliminated ARM emulation and leveraged virtualization technologies built-in to CPUs. This was better. In fact, it was so much better that a lot of people were satisfied — myself included.</p>
<p>Recently, a company called <a href="http://www.genymotion.com/">Genymotion</a> was formed around the work of a project that compiled Android to run in a VirtualBox VM. They not only delivered an experience that was faster and simpler than the x86 emulator, but they provided much-needed tools for modern app development. Easier sensor controls, touchscreen input using a remote device, and screen capture support for both images and video are just some of these features which really make the product shine.</p>
<p>Hopefully none of what I’ve covered is news to you. But now I am ready to start talking about this post’s true topic:</p>
<p><img src="/static/post-image/avd-manager.png" width="300" class="pull-right" /></p>
<p><strong>All existing emulator solutions are terrible.</strong></p>
<p>The management interface for creating, configuring, and starting emulators is a minimumly-viable Swing app. While the low quality list view is actually fine, the configuration pane is a mess <em>(pictured)</em>.</p>
<p>This screen lacks any design to facilitate the correct behavior. The instruction set is usually defaulted to ARM which has to be fully emulated (very slow). Use of the host GPU is defaulted to off which means that the display pipeline will not be hardware accelerated (again, very slow). Unless you’ve been Googling passive-aggressive phrases about the emulator speed you might never even cross <a href="https://software.intel.com/en-us/android/articles/intel-hardware-accelerated-execution-manager">HAXM</a>.</p>
<p>The pain does not cease once the emulator is running (whether using optimal settings or not). Each instance requires significant system resources and the performce of the contained OS will vary wildly. Instances will occasionally hang, crash, or disappear from <code class="highlighter-rouge">adb</code>’s visibility requiring manual restarts. Controls and interfaces to a few of the sensor are present, but they are far from comprehensive.</p>
<p>Genymotion is a step up from the first-party offering. It has fewer options for configuration because it is already set up for optimal performance. As previously mentioned, its sensor and developer controls are much more rich and useful.</p>
<p>The initial downside of Genymotion is the required user account and strange pricing of commercial licenses (and lack of a site license). They also are not without problems which plague the actual use of the emulator. The VirtualBox images occasionally get corrupted or stuck which require a trip in the depths of your filesystem for manual purging. The free license cripples functionality that would otherwise exist if they hadn’t explicitly disabled it (screenshot, recording). Their pricing model for commercial use also does not reflect the amount of utility you actually receive.</p>
<p>We put up with these solutions because they are an improvement compared to what came prior. However, they are nowhere near what we truly need or deserve.</p>
<hr />
<p><strong>Android needs a simulator for day-to-day development and testing.</strong></p>
<blockquote>
<p>sim·u·la·tor <em>/ˈsimyəˌlātər/</em></p>
<p>A machine with a similar set of controls designed to provide a realistic imitation of the operation of a vehicle, aircraft, or other complex system, used for training purposes.</p>
</blockquote>
<p>A simulator is a shim that sits between the Android operating system runtime and the computer’s running operating system. It bridges the two into a single unit which behaves closely to how a real device or full emulator would at a fraction of the overhead.</p>
<p>The most well known simulator to any Android developer is probably (and ironically) the one that iOS develoers use from Apple. The iPhone and iPad simulators allow quick, easy, and lightweight execution of in-development apps. If you haven’t seen this simulator in action, I would encourage you to take a two-minute tour of one before continuing this post.</p>
<p>What does a simulator buy us that a traditional emulator does not?</p>
<ul>
<li>Creating, configuration, and running simualtors becomes about the runtime, not the right configuration of options for optimal performance.</li>
<li>Instrumentation tests gain stability and speed which thus massively increases their utility. A simulator that runs headless as part of the normal build means your build can run these tests no differently than it compiles the Java sources.</li>
<li>The need for a separate, JVM-based unit test solution diminishes drastically. Even more exciting is that the need for a third-party testing solution like Robolectric dimishes with it. When a headless simulator is part of your build (and with an <a href="https://android.googlesource.com/platform/frameworks/testing/+/master/androidtestlib/">upcoming test runner diversification</a>), unit tests on the JVM become a first-party delight.</li>
<li>The lack of an emulated architecture layer, the overhead of a whole OS running, and the need for build steps like packaging (more on that to follow) means you are able to develop and deploy at a speed which just isn’t possible in the current setup.</li>
</ul>
<p>There always will be a need for a proper emulator for acceptance testing your application in an environment that behaves exactly like a device. For day-to-day development this is simply not needed. Developer productivity will rise dramatically and the simplicity through which testing can now be done will encourage their use and with any luck improve overall app quality.</p>
<hr />
<p>Android actually already has two simulators which are each powerful in different ways, but nowhere near powerful enough. Before we talk about them, let’s cover why a simulator is a perfect fit for Android development.</p>
<p>Apps are text files of Java The Language™, compiled with <code class="highlighter-rouge">javac</code> to JVM bytecode, transformed with <code class="highlighter-rouge">dex</code> to Dalvik bytecode, zipped up into an <code class="highlighter-rouge">.apk</code>, and signed using <code class="highlighter-rouge">jarsigner</code>. Other tools like <code class="highlighter-rouge">zipalign</code> and ProGuard are optionally a part of this toolchain but since they aren’t usually used in development we can safely ignore them. Prior to the invocation of <code class="highlighter-rouge">javac</code>, all of the resources of an app must be parsed with <code class="highlighter-rouge">aapt</code> for code generation and special encoding of some files. This is a lot of steps!</p>
<p>Using a simulator would reduce this to a single step: compilation with <code class="highlighter-rouge">javac</code>. We are already relying on the JVM for running our IDE, our build system, and our compilation. Why aren’t we leveraging it for running a simulated OS?</p>
<p>Ok so I glossed over two other toolchain components we’d need to run our class files in a JVM-hosted simulator:</p>
<ol>
<li>A modified <code class="highlighter-rouge">aapt</code> whose only responsibility was generation of <code class="highlighter-rouge">R</code> files would still be needed. Thankfully the slower operations this resource step performs (image optimization and text file encoding) wouldn’t be needed. XML files can be read on-the-fly as text by the simulator. Images don’t need optimized since they are just being displayed from the local filesystem.</li>
<li>A signing key is required for the OS to verify installation and grant special permissions. Rather than having to actually sign anything, a simple certificate can be created from the keystore and included as a string.</li>
</ol>
<p>Imagine how quick the time between modifying your source code and running the application becomes when the only steps needed are a resource scan, <code class="highlighter-rouge">javac</code>, and copying a string. Oh, but do you have a ton of dependencies? Not a problem since all that’s needed is appending the file path of the <code class="highlighter-rouge">.jar</code> file onto the JVM classpath.</p>
<p>The prospect of this “exploded” <code class="highlighter-rouge">.apk</code> application should get you seriously excited.</p>
<p>Even more exciting is that there are already two simulators which work with these exploded apps. The first and most well known is <a href="http://robolectric.org/">Robolectric</a>, a tool for running unit tests on the JVM. The second is named “<a href="https://android.googlesource.com/platform/frameworks/base/+/master/tools/layoutlib">layoutlib</a>” which is far less known but is used daily by every Android developer.</p>
<ol>
<li>
<p>Robolectric runs a compiled version of the OS in a separate classloader using techniques like bytecode rewriting and proxies. This puts most of the real OS infrasture at your disposal for unit testing code paths of your app that have to touch Android code. People often abuse Robolectric for testing the wrong things but it usually works because the real OS code is used by default.</p>
<p>It takes about two seconds for Robolectric to initialize. Most of this time is creating the custom classloader and initializing all the proxy classes. Once running application code is loaded into the classloader and run like normal Java code. The resources are lazily resolved directly from the source files.</p>
</li>
<li>
<p>“layoutlib” is a module whose purpose is to run view code on the JVM including parsing layout XML and loading resources. If you’ve ever used the layout designer or layout preview in either Eclipse or IntelliJ IDEA/Android Studio then you have used this library.</p>
<p>Running your view code (including custom views) is done like any other Java code. The classes inside the library fake out the <code class="highlighter-rouge">Context</code> and the resource loading it brings. The rendering pipeline is also simply mapped into normal Java rendering primitives so you can see real-time updates of your layouts.</p>
</li>
</ol>
<p>Both of these libaries use very clever techniques to simulate parts of Android to great success. Neither one is suited to running an application during development which is what we are after.</p>
<hr />
<p>There are hurdles to be tackled in building a simulator that can host development applications. If you’ll remember from above, we already have a pared down version of <code class="highlighter-rouge">aapt</code>, a simple representation of the signing key, and are leveraging <code class="highlighter-rouge">javac</code> and the JVM classpath for loading code and libraries. Let’s enumerate what else is required — none of which are insurmountable.</p>
<ul>
<li>Native code has to be compiled for x86 in order to be run. Aside from the regular pain of JNI there should not be too much trouble here.</li>
<li>The graphics pipeline of the OS needs hooked in to the host. I won’t pretend to know a lot about what would be required in this area. Native code covers software rendering and hooking up OpenGL should facilitate hardware rendering.</li>
<li>A variety of interfaces with hardware need replicated or faked. Some have obvious native equivalents on the host like bluetooth and networking. Some can simply be fixed to default values like the accelerometer and compass. Others can not be marked as present or just emulated.</li>
</ul>
<p>Thankfully these are all solvable problems. Each one just needs the right person with the time and effort to tackle it. However, therein lies another problem.</p>
<p>The single greatest hurdle to the creation of the simulator we deserve is the time, effort, and desire required to build and maintain it. Sorry, tools team! I’m told <a href="https://twitter.com/droidxav/status/476181900357169152">they’re hiring</a>.</p>
<p><em>Follow the discussion on <a href="https://plus.google.com/108284392618554783657/posts/FYxnhQuEyoq">Google+</a> and <a href="http://www.reddit.com/r/androiddev/comments/28al7i/android_needs_a_simulator_not_an_emulator/">Reddit</a>.</em></p>
Hello Picasso 2.32014-05-30T00:00:00+00:00https://jakewharton.com/hello-picasso-2.3This post was published externally on Square Corner. Read it at https://developer.squareup.com/blog/hello-picasso-2-3.Dynamic Images with Thumbor2014-01-20T00:00:00+00:00https://jakewharton.com/dynamic-images-with-thumborThis post was published externally on Square Corner. Read it at https://developer.squareup.com/blog/dynamic-images-with-thumbor.Enhance Your Application Using Picasso2013-05-14T00:00:00+00:00https://jakewharton.com/enhance-your-application-using-picassoThis post was published externally on Square Corner. Read it at https://developer.squareup.com/blog/enhance-your-application-using-picasso.Easy HTTP Requests with Retrofit2013-05-13T00:00:00+00:00https://jakewharton.com/easy-http-requests-with-retrofitThis post was published externally on Square Corner. Read it at https://developer.squareup.com/blog/easy-http-requests-with-retrofit.MimeCraft, JavaWriter, and ProtoParser2013-05-08T00:00:00+00:00https://jakewharton.com/mimecraft-javawriter-and-protoparserThis post was published externally on Square Corner. Read it at https://developer.squareup.com/blog/mimecraft-javawriter-and-protoparser.Seven Days of Open Source2013-05-06T00:00:00+00:00https://jakewharton.com/seven-days-of-open-sourceThis post was published externally on Square Corner. Read it at https://developer.squareup.com/blog/seven-days-of-open-source.The Resurrection of Testing for Android2013-04-03T00:00:00+00:00https://jakewharton.com/the-resurrection-of-testing-for-androidThis post was published externally on Square Corner. Read it at https://developer.squareup.com/blog/the-resurrection-of-testing-for-android.Deprecated From Inception2012-10-01T00:00:00+00:00https://jakewharton.com/deprecated-from-inception<p>Stop using ActionBarSherlock! Well… eventually.</p>
<p>If you are writing an application right now with a <code class="highlighter-rouge">minSdkVersion</code> lower than 14 you should be using it. I am not saying this just as the developer of the library, but as someone who likes to minimize wasted time.</p>
<p>Writing applications is hard and nobody wants to spend time writing boilerplate code. It sucks valuable engineering time away from what is most important in your app: the content. Throw in ActionBarSherlock and a slew of other open source libraries to bootstrap your development so you can make the best app possible.</p>
<p>In order to have pleasant longevity, libraries you integrate into your application should be as small and modular as possible. Inevitably new libraries will be released and some will have features that you wish to integrate at the cost of others. If a library is small, these replacements can be seamless (and often times drop-in).</p>
<p>ActionBarSherlock is very much the opposite of a small, modular library. When you decide to include ActionBarSherlock in your app you are committing to using it for the foreseeable future. Or are you?</p>
<p>By design, ActionBarSherlock uses the native action bar API and theming. Yes you have to use types that exist in different packages and duplicate theme attributes but the names of classes, methods, and attributes mirror their native counterparts exactly. In fact, you have to go out of your way in order to find things that are not API-compatible with the native action bar.</p>
<p>Some day, market saturation and the demands of your application will necessitate updating your <code class="highlighter-rouge">minSdkVersion</code> to 14 (or higher).</p>
<p>If you are using a different action bar library or you have rolled your own then you will either have zero work ahead of you (which means you will still be using your crappy implementation) or you have a lot of work ahead of you (migrating to use the native components). Both of these situations are not ideal and I can think of handfuls of massively popular apps who will someday find themselves in this predicament.</p>
<p>On the other hand, if you happen to be using ActionBarSherlock, all you have to do is switch from using the custom types back to the native types. This switch is so easy that 99% of it can be scripted. It boils down to changing a few imports, replacing calls to <code class="highlighter-rouge">getSupportActionBar</code> with <code class="highlighter-rouge">getActionBar</code>, and using a <code class="highlighter-rouge">Holo</code> parent theme rather than a <code class="highlighter-rouge">Sherlock</code> one.</p>
<p><strong>This is the entire purpose of the library. It has been designed for this use case specifically. It has been deprecated from inception with the intention that you could some day throw it away.</strong></p>
<p>The process also works in reverse, in fact. At AnDevCon III I gave a talk where I migrated multiple code examples written for the native action bar to work with ActionBarSherlock. The entire process took less than a few minutes and the result was code that only ran on ~8% of devices (at the time) now being able to be run on about 91% of devices.</p>
<p>Google has announced that they are working on a library that will backport a subset of the action bar. I was disappointed in this anouncement for two reasons: it was announced 12 months too late (and remains yet unreleased) and it is wasting engineering time of a talented developer. There are so many gaping holes in Android development where their time could be better spent. If they wanted to come out with a backport then it should have been done at the same time as ICS dropped.</p>
<p>Let us hope they honor the notion of deprecation from inception as well, otherwise they will only make things worse.</p>
<p><em>Follow the discussion on <a href="https://plus.google.com/u/1/108284392618554783657/posts/SA1KF2uHBnM">Google+</a> and <a href="http://www.reddit.com/r/androiddev/comments/10rybj/actionbarsherlock_deprecated_from_inception/">Reddit</a>.</em></p>
The Android Build System Is Broken2012-07-22T00:00:00+00:00https://jakewharton.com/the-android-build-system-is-broken<p>The blessed souls in the Android world with regard to compilation are Eclipse
and ant. Both serve admirably if you have a small-to-medium-sized app. You might
even pull in a library project and a jar or two. This works great. As the
complexity grows beyond this, however, both of these players break down and you
are left to fend for yourself.</p>
<p>Advanced configurations which are designed to make it more efficient for you,
the developer, end up causing unnecessary strain because the build system
cannot handle it. When you have multiple modules for production and
development versions of your app so that both can be installed at once and
library projects that depend on library projects that depend on library
projects—all of which have different overlapping jar
dependencies—you are on your own.</p>
<p>Why is this? Well it is mostly because I have been lying to you. <strong>Android does
not have a build system.</strong></p>
<p>What Android has is a scripting language that has been shoved unceremoniously
into XML, a default configuration that attempts to cover all of your use cases,
and an IDE whose configuration attempts to mirror the scripting language
configuration but has only marginal integration.</p>
<p>This is awful.</p>
<p>The Android community should settle for nothing short of the following:</p>
<ol>
<li><em>Dependency management</em> - You should never have to copy a jar or library
project into your tree. Version number differences should automatically
be resolved. Transitive dependencies should be recursively pulled in.</li>
<li><em>Build order</em> - Multi-module builds are a directed, acyclic graph and the
order of their compilation can be determined by the build system. If you
add a dependency between two modules the order should automatically change
to accommodate.</li>
<li><em>Non-Android projects</em> - Modules should not have to be Android library
projects to be part of the build path. Pure Java projects (and anything
that compiles to class files) must be supported.</li>
<li><em>Seamless IDE integration</em> - Changes to configurations should be reflected
in both command-line builds and IDE builds without any additional effort.</li>
</ol>
<p>Could all of this be accomplished with ant and Eclipse? Maybe. Should it be
attempted? Absolutely not.</p>
<p>I use maven and IntelliJ IDEA for all of my projects and while it solves all of
the requirements listed above, it does not feel perfect. At Square we are
currently using ant (with a lot of customization) and IntelliJ IDEA. I think
almost everyone on the team would agree that it feels far from perfect but it
works well enough. Results are hard to argue with but a resounding endorsement
this is not.</p>
<p>A build system should empower, not constrain. It should enable, not restrict. It
should be dynamic, not rigid.</p>
<p>The bottom line is that whether you use maven, sbt, or gradle we all lose
because Google is advocating and supporting ant and the Eclipse plugin.</p>
<p>We finally have an operating system that has been refined at an amazing level of
detail. We have tooling around developing and debugging applications to an
unparalleled depth. We deserve a build system with the same attention to detail.</p>
<p><em>Follow discussion on <a href="https://plus.google.com/108284392618554783657/posts/KuWYBtLKtSE">Google+</a> or <a href="http://www.reddit.com/r/androiddev/comments/wztec/the_android_build_system_is_broken/">Reddit</a></em></p>
<p><strong>Update:</strong> Be sure you check out Xavier Durochet’s reply on the Google+ thread
(approximately 17 comments down).</p>
Decoupling Android App Communication with Otto2012-07-02T00:00:00+00:00https://jakewharton.com/decoupling-android-apps-with-ottoThis post was published externally on Square Corner. Read it at https://developer.squareup.com/blog/decoupling-android-app-communication-with-otto.Using ActionBarSherlock As A Base2012-05-15T00:00:00+00:00https://jakewharton.com/using-actionbarsherlock-as-a-baseThis post was published externally on Square Corner. Read it at https://developer.squareup.com/blog/using-actionbarsherlock-as-a-base.Announcing ActionBarSherlock Version 4.02012-03-07T00:00:00+00:00https://jakewharton.com/actionbarsherlock-four-point-oh<p>It’s been approximately three months since the Android 4.0 Ice Cream Sandwich source code landed on the Android Open Source Projects’s servers. Since then I have spent countless nights developing what I described in an <a href="/actionbarsherlock-a-love-story-part-1">earlier blog post</a> as the “first true release” of ActionBarSherlock: version 4.0. As of 11:57:57PM PST I have finally tagged and released this new revolutionary version of the library into the wild for your consumption.</p>
<p>This date of release was chosen because of its historical significance within the scope of ActionBarSherlock. Exactly one year ago the first version, 1.0, was tagged and released. If you read the <a href="/actionbarsherlock-a-love-story-part-2">history lesson blog post</a> you’ll know that this version only lasted 24 hours before 2.0 came along but it still represents the very first milestone. Version 4.0 is another huge milestone so the fact that it is occurring on the same date should help convey its significance.</p>
<p>For those who are not aware, version 4.0 is a feature-complete backport of the Android 4.0 action bar and its supporting widgets to Android 2.x and up. This allows applications to integrate a 100% API and theme-compatible action bar with extremely little effort. Rather than worrying about interfacing with a custom third-party action bar or adding features to the ActionBarCompat sample, developers can now drop in ActionBarSherlock and focus on the most important part of their app, the content!</p>
<p>In the next 48 hours I plan to fill out the release a bit with a proper migration guide for applications coming from the v3.x branch along with some minor updates to the <a href="http://actionbarsherlock.com/">website</a> with new screenshots. Savy developers should already have all they need, however. The library is <a href="http://actionbarsherlock.com/download.html">available for download</a> on the website along with its sample applications–the source code to which is included in the download.</p>
<p>For those who would like to stay up-to-date with the library in a more automated fashion you can download its samples from the newly re-branded Play Store:</p>
<ul>
<li><a href="https://play.google.com/store/apps/details?id=com.actionbarsherlock.sample.demos">ActionBarSherlock: Demos Sample</a></li>
<li><a href="https://play.google.com/store/apps/details?id=com.actionbarsherlock.sample.fragments">ActionBarSherlock: Fragments Sample</a></li>
<li><a href="https://play.google.com/store/apps/details?id=com.actionbarsherlock.sample.roboguice">ActionBarSherlock: RoboGuice Sample</a></li>
</ul>
<p>I would like to issue a special thank you to all the developers with whom I have worked closely with on the development of this release and for all of the detailed feedback, bug reporting, and (my personal favorite) pull requests. Some of these developers even have already released applications which are using early versions of the library and I urge you to show them your support as well:</p>
<ul>
<li><a href="https://play.google.com/store/apps/details?id=com.battlelancer.seriesguide">SeriesGuide</a> (well, the <a href="https://play.google.com/store/apps/details?id=com.battlelancer.seriesguide.beta">beta</a> is)</li>
<li><a href="https://play.google.com/store/apps/details?id=com.github.mobile.gauges">GitHub Gaug.es</a></li>
<li><a href="https://play.google.com/store/apps/details?id=com.androiduipatterns.mentionobserver">Mentions</a></li>
</ul>
<p>If you release an application that uses ActionBarSherlock version 4 I would love to hear about it. Please contact me via <a href="http://twitter.com/JakeWharton">Twitter</a>, <a href="http://profiles.google.com/jakewharton">Google+</a>, or <a href="mailto:jakewharton@gmail.com">email</a>.</p>
<hr />
<p><em>Quick follow up:</em> It should be noted that some bugs still exist and may rear their heads depending on how you use the library. The library is under constant development and I will do my best to get updates and fixes out ASAP. The library is certainly more stable than not and I could not in good conscience allow anyone to develop using v3.5.x any longer. I’m hoping that the increased exposure that should come with the final release will help attract more developer attention which should aid in resolving bugs more quickly.</p>
<p><strong>Please use the Google group for all library-related discussions @ <a href="http://abs.io/forum">abs.io/forum</a></strong></p>
<p>Also, pull requests save lives so send them!</p>
<p> </p>
<p><img src="/static/post-image/actionbarsherlock-logo.png" alt="ActionBarSherlock Logo" /> <img src="/static/post-image/forever-alone-birthday.jpg" alt="Happy Birthday to me!" /> <img src="/static/post-image/android-ice-cream-sandwich.jpg" alt="Ice Cream Sandwich" /></p>
Advanced Pre-Honeycomb Animation with NineOldAndroids2012-01-18T00:00:00+00:00https://jakewharton.com/advanced-pre-honeycomb-animation<p>The lovely new <a href="http://android-developers.blogspot.com/2011/02/animation-in-honeycomb.html">animation framework</a> in Android 3.0 came with some additional
methods on the <code>View</code> class to allow for transformations such as
translation, scale, rotation, and alpha. The <a href="http://nineoldandroids.com">NineOldAndroids</a> library allows
for the use of this API on any platform but is limited only to modifying values
for which methods exist on the running platform.</p>
<p>Recently I <a href="http://stackoverflow.com/q/8734001/132047">set out</a> to solve this problem and allow for utilizing my library
to animate these properties regardless of the API level. Neither an answer to
the linked StackOverflow question nor a quick exchange with the animation guru
Chet Haase himself semed to produce a reliable, stable implementation for
this–the recommendation always being to just use the built-in view animation.</p>
<p>As I was digging around in the <code class="highlighter-rouge">View</code> class I noticed that there really was no
way to achieve this effect directly, even with reflection. It was only once I
started poking around how view animations are processed and executed did a
rather clever solution appear to me.</p>
<p>For those that are not familiar with how the view animation framework works,
an animation receives a callback with a <code class="highlighter-rouge">Transformation</code> object and a time
interval. Each animation then adjusts the object in order to reflect the state
of the associated view at whatever time interval it is at. Since the
<code class="highlighter-rouge">Transformation</code> object contains a method for setting alpha and a matrix which
is applied to the canvas rendering the view we can easily achieve all of the
transformations of the native methods introduced in Honeycomb.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">void</span> <span class="nf">applyTransformation</span><span class="o">(</span><span class="kt">float</span> <span class="n">interpolatedTime</span><span class="o">,</span> <span class="nc">Transformation</span> <span class="n">t</span><span class="o">)</span> <span class="o">{</span>
<span class="c1">//Perform transformations</span>
<span class="o">}</span>
</code></pre></div></div>
<p>So now that we know these transformations were possible, how best to implement
them in a manner that can be used by the new animation API? To accomplish this
we use a few tricks of view animation in order to do this in a way that is as
lightweight and fast as possible (we’re on the UI thread, remember).</p>
<p>Only one animation can be applied to a view at a time so it was obvious that a
custom class extending <code class="highlighter-rouge">Animation</code> was required to apply our many
transformations. Now it became a matter of synchronizing our new class with the
NineOldAndroids library since it would be the one actually controlling the
animation.</p>
<p>Instead of attempting to integrate NineOldAndroids directly in this custom class
I chose to make it act only as a proxy to the alpha and the <code class="highlighter-rouge">Transformation</code>
object by exposing methods to allow changing the various properties that were
introduced in Honeycomb.</p>
<p>In order to take the native stepping of view animation out of the equation, our
custom class immediately sets two properties on itself: <code class="highlighter-rouge">setDuration(0)</code> and
<code class="highlighter-rouge">setFillAfter(true)</code>. This effectively disables the timer internally triggering
the transformation and it allows the transformations that we make to be
persisted on the view after the animation has completed. In order for the latter
to occur the animation is kept around so that its transformation can be applied
whenever the view is invalidated. This is the behavior that we leverage in order
to provide our animation.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">AnimatorProxy</span><span class="o">(</span><span class="nc">View</span> <span class="n">view</span><span class="o">)</span> <span class="o">{</span>
<span class="n">setDuration</span><span class="o">(</span><span class="mi">0</span><span class="o">);</span> <span class="c1">//perform transformation immediately</span>
<span class="n">setFillAfter</span><span class="o">(</span><span class="kc">true</span><span class="o">);</span> <span class="c1">//persist transformation beyond duration</span>
<span class="n">view</span><span class="o">.</span><span class="na">setAnimation</span><span class="o">(</span><span class="k">this</span><span class="o">);</span>
<span class="n">mView</span> <span class="o">=</span> <span class="n">view</span><span class="o">;</span>
<span class="o">}</span>
</code></pre></div></div>
<p>We expose our new properties as getter and setter methods that the new animation
API can interact with and hold them in instance variables in our animation. Each
invalidation then triggers our callback which we can then apply the newly
updated values for each property, thus, animating the view.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">void</span> <span class="nf">setAlpha</span><span class="o">(</span><span class="kt">float</span> <span class="n">alpha</span><span class="o">)</span> <span class="o">{</span>
<span class="n">mAlpha</span> <span class="o">=</span> <span class="n">alpha</span><span class="o">;</span>
<span class="n">mView</span><span class="o">.</span><span class="na">invalidate</span><span class="o">();</span>
<span class="o">}</span>
</code></pre></div></div>
<p>This works extremely well and provides fluid, multi-property animation using
NineOldAndroids for the new animation API but it still requires us to use the
animation proxy class for these specific properties. In order to provide a more
seamless experience, we need a way to have this handled automatically.</p>
<p>In order to determine when this class is required we add a small check in the
initialization method of <code class="highlighter-rouge">ObjectAnimator</code>. If the animation meets the following
four conditions then a proxy instance is used: we are using a named property and
not a <code class="highlighter-rouge">Property</code>, we are running on pre-3.0 Android, the target class is an
instance of <code class="highlighter-rouge">View</code>, and the named property is one of the ones introduced in
Honeycomb.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="o">((</span><span class="n">mProperty</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span> <span class="o">&&</span> <span class="nc">AnimatorProxy</span><span class="o">.</span><span class="na">NEEDS_PROXY</span> <span class="o">&&</span> <span class="o">(</span><span class="n">mTarget</span> <span class="k">instanceof</span> <span class="nc">View</span><span class="o">)</span>
<span class="o">&&</span> <span class="no">PROXY_PROPERTIES</span><span class="o">.</span><span class="na">containsKey</span><span class="o">(</span><span class="n">mPropertyName</span><span class="o">))</span> <span class="o">{</span>
<span class="n">setProperty</span><span class="o">(</span><span class="no">PROXY_PROPERTIES</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">mPropertyName</span><span class="o">));</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Here, <code class="highlighter-rouge">PROXY_PROPERTIES</code> is a <code class="highlighter-rouge">Map</code> which maps the required property names to
special <code class="highlighter-rouge">Property</code> classes that automatically use an instance of our proxy
animation class. By setting a <code class="highlighter-rouge">Property</code> instance on the animation we will
essentially override the string equivalent so that reflection on the method
is not attempted.</p>
<p>Now you can enjoy advanced Honeycomb-style animation of post-Honeycomb <code class="highlighter-rouge">View</code>
properties by simple changing your imports to use NineOldAndroids!</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">AnimatorSet</span> <span class="n">set</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">AnimatorSet</span><span class="o">();</span>
<span class="n">set</span><span class="o">.</span><span class="na">playTogether</span><span class="o">(</span>
<span class="nc">ObjectAnimator</span><span class="o">.</span><span class="na">ofFloat</span><span class="o">(</span><span class="n">myView</span><span class="o">,</span> <span class="s">"rotationX"</span><span class="o">,</span> <span class="mi">0</span><span class="o">,</span> <span class="mi">360</span><span class="o">),</span>
<span class="nc">ObjectAnimator</span><span class="o">.</span><span class="na">ofFloat</span><span class="o">(</span><span class="n">myView</span><span class="o">,</span> <span class="s">"rotationY"</span><span class="o">,</span> <span class="mi">0</span><span class="o">,</span> <span class="mi">180</span><span class="o">),</span>
<span class="nc">ObjectAnimator</span><span class="o">.</span><span class="na">ofFloat</span><span class="o">(</span><span class="n">myView</span><span class="o">,</span> <span class="s">"rotation"</span><span class="o">,</span> <span class="mi">0</span><span class="o">,</span> <span class="o">-</span><span class="mi">90</span><span class="o">),</span>
<span class="nc">ObjectAnimator</span><span class="o">.</span><span class="na">ofFloat</span><span class="o">(</span><span class="n">myView</span><span class="o">,</span> <span class="s">"translationX"</span><span class="o">,</span> <span class="mi">0</span><span class="o">,</span> <span class="mi">90</span><span class="o">),</span>
<span class="nc">ObjectAnimator</span><span class="o">.</span><span class="na">ofFloat</span><span class="o">(</span><span class="n">myView</span><span class="o">,</span> <span class="s">"translationY"</span><span class="o">,</span> <span class="mi">0</span><span class="o">,</span> <span class="mi">90</span><span class="o">),</span>
<span class="nc">ObjectAnimator</span><span class="o">.</span><span class="na">ofFloat</span><span class="o">(</span><span class="n">myView</span><span class="o">,</span> <span class="s">"scaleX"</span><span class="o">,</span> <span class="mi">1</span><span class="o">,</span> <span class="mf">1.5f</span><span class="o">),</span>
<span class="nc">ObjectAnimator</span><span class="o">.</span><span class="na">ofFloat</span><span class="o">(</span><span class="n">myView</span><span class="o">,</span> <span class="s">"scaleY"</span><span class="o">,</span> <span class="mi">1</span><span class="o">,</span> <span class="mf">0.5f</span><span class="o">),</span>
<span class="nc">ObjectAnimator</span><span class="o">.</span><span class="na">ofFloat</span><span class="o">(</span><span class="n">myView</span><span class="o">,</span> <span class="s">"alpha"</span><span class="o">,</span> <span class="mi">1</span><span class="o">,</span> <span class="mf">0.25f</span><span class="o">,</span> <span class="mi">1</span><span class="o">)</span>
<span class="o">);</span>
<span class="n">set</span><span class="o">.</span><span class="na">setDuration</span><span class="o">(</span><span class="mi">5</span> <span class="o">*</span> <span class="mi">1000</span><span class="o">).</span><span class="na">start</span><span class="o">();</span>
</code></pre></div></div>
<p>Download NineOldAndroids 2.0.0 from <a href="http://nineoldandroids.com">nineoldandroids.com</a> and check it out
on <a href="http://github.com/JakeWharton/NineOldAndroids/">GitHub</a>.</p>
Something Beta This Way Comes!2012-01-02T00:00:00+00:00https://jakewharton.com/something-beta-this-way-comes<p>Some implementation details…</p>
<p>There are 6 base activities, 4 of which are the core library and one in each of two plugins.</p>
<ul>
<li>Core: <code class="highlighter-rouge">SherlockActivity</code>, <code class="highlighter-rouge">SherlockPreferenceActivity</code>, <code class="highlighter-rouge">SherlockListActivity</code>, and <code class="highlighter-rouge">SherlockExpandableListActivity</code> - Former two should be obvious, latter two shouldn’t really be used anymore. Try fragments.</li>
<li>Compat-Lib: <code class="highlighter-rouge">FragmentActivity</code> - Modified version of the official library to support having an action bar by default.</li>
<li>Maps: <code class="highlighter-rouge">SherlockMapActivity</code> - Since referencing the base class MapActivity requires compiling with the Google APIs this is in its own plugin. Remember: This does not work like <code class="highlighter-rouge">FragmentMapActivity</code> from v3.x.</li>
</ul>
<p>Everything has been moved into the <code class="highlighter-rouge">com.actionbarsherlock.*</code> package tree. Stay away from <code class="highlighter-rouge">com.actionbarsherlock.internal.*</code>. Check your imports.</p>
<p>Again, as with v3.x, the default options menu methods have been marked final in the base activities so that you cannot use them erroneously. A majority of the supposed bugs that I receive have to do with incorrect imports. Check your imports and check the samples before filing a bug or emailing the mailing list.</p>
<p>Don’t file a bug or suggestion on anything related to the compat-lib plugin that does not directly relate to this library. As nice as it was to upstream some bugfixes I am not doing that anymore. File them on <a href="http://b.android.com">b.android.com</a>.</p>
<p><code class="highlighter-rouge">SherlockPreferenceActivity</code> does not have fragment or loader support like v3.x did. No I will not enable it. Yes I’ll look at porting <code class="highlighter-rouge">PreferenceFragment</code> in the future. Don’t ask for an ETA.</p>
<p>There are bugs and missing features. Check <a href="http://abs.io/4">abs.io/4</a> before reporting anything! Check the samples. If a sample is missing, take a few minutes to write it.</p>
<p>Pay attention for new betas. Check the website often. You can even follow the site’s repository on GitHub for better notification: <a href="https://github.com/JakeWharton/beta.abs.io">github.com/JakeWharton/beta.abs.io</a>.</p>
<p>Want fixes sooner? Check the <a href="http://abs.io/b/4.0-wip"><code class="highlighter-rouge">4.0-wip</code></a> branch. You’ll have to build the plugins yourself though if changes were made. Please don’t ask me how. Maven, SDK deployer, and mvn clean package.</p>
<p>Try everything. Write a new app, port an old app, write more samples. Do something. Don’t complain if you jump on the final release and find bugs without having trying the betas.</p>
<p>Use <code class="highlighter-rouge">Theme.Sherlock</code>. Use <code class="highlighter-rouge">Theme.Sherlock</code>!</p>
<p>There is no light theme… yet. Use black for testing. Don’t complain and don’t bother implementing it. A light and a light/dark action bar theme will be present in the release candidate.</p>
<p>Things are broken. Most is working. Try before you buy. All sales are final.</p>
<p>I’ll leave you with a semi-related, partially-humorous quote from Equilibrium (which is actually from W. Yeats)</p>
<blockquote>
<p>But I, being poor, have only my dreams. I have spread my dreams under your feet. Tread softly because you tread on my dreams.</p>
</blockquote>
<p><strong>How to report bugs:</strong></p>
<p>Fix it yourself and send a pull request…</p>
<p>Ok, you don’t <em>have</em> to do that but I’ll seriously love you for it.</p>
<p>Create a new issue on GitHub, include as much description, code, and images as humanly possible to make your problem apparent to someone who has never done Android. It’s not that don’t understand your problem, it’s that I don’t want to have to spent extra time deciphering it or have any doubt about what you think the problem is.</p>
<p>I’ll do my best to thank you no matter how severe of a bug you find :)</p>
ActionBarSherlock - A Love Story (Part 3)2012-01-01T00:00:00+00:00https://jakewharton.com/actionbarsherlock-a-love-story-part-3<p>I am talking, of course, about version 3 and version 4, respectively. And I’m also lying a bit because I won’t just be abandoning the 3.x users either. I’ll give you a two-month deprecation window from version 4’s release. Because…</p>
<p><strong>ActionBarSherlock v4 is coming and it is awesome.</strong></p>
<p>Now I realize that I am a bit biased, but let me explain how this version is the first version that I think I will be truly proud of.</p>
<ol>
<li>
<p>No more shuffling between native and custom implementations.</p>
<p>Google’s support library operates in this way in that it makes no attempt to use any native implementations even if they exist. It is far easier and more stable to keep all of the functionality in the library. Plus, the Android 4.0 action bar has been designed to accommodate every conceivable screen size that the platform can run on so why should we continue bothering to switch to the native implementation?</p>
<p>Additionally, this change became more and more of an apparent need rather than a choice due to changes in Android 4.0’s <code class="highlighter-rouge">MenuItem</code> interface.</p>
</li>
<li>
<p>The support library classes are no longer included in the core of the library.</p>
<p>Though somewhat ironic based on the last point, the decision to allow the library to stand alone was made in order to accommodate developers who were uncomfortable using a custom built version of the support library (or who even didn’t use it at all).</p>
<p>A version of the support library will be provided as a plugin .jar that has been modified to add ActionBarSherlock support. The changes to the library will be kept at a minimum and will not include any unrelated fixes. File bugs on b.android.com for that, please.</p>
<p><strong>WARNING:</strong> This means that if you are using <code class="highlighter-rouge">FragmentMapActivity</code> or using fragments in <code class="highlighter-rouge">SherlockPreferenceActivity</code> you WILL have to change your implementation or create your own versions of these base classes. I will no longer be maintaining support for these.</p>
</li>
<li>
<p>Extending from a custom base activity is no longer required (but still recommended).</p>
<p>Similar to how ActionBarSherlock v1 and v2 operated, you can perform static attachment of the action bar to your activities. This allows for the use of alternate base activities such as those provided by other third-party libraries (e.g., RoboGuice).</p>
<p>The added side-effect of this is that all of the interaction logic has been placed within this single class which is also the one used by the base activities. This means that whether you do use a base activity or choose to interact with the static attachment you are afforded the full API.</p>
</li>
<li>
<p>Fully mirrored theming support to mimic the native action bar.</p>
<p>Forget the ‘ab’-prefixed attributes of v3.x, v4 now allows for defining proper styles for the action bar, action mode, and various other sub-components of the action views.</p>
</li>
<li>
<p>It is the Ice Cream Sandwich action bar!</p>
<p>…but you probably knew that already.</p>
<p>Split action bar, action modes, action providers, condensed tab navigation, and so much more!</p>
</li>
</ol>
<p>I have been working on this for nearly 8 weeks now so it’s easy for me to get excited. Starting tomorrow the version 4 beta will be officially announced and detailed in a much more technical manner so that you can begin testing and hopefully join in the excitement.</p>
<p>As it stands now there are still large bugs and “bugs” with version 4. You can find them <a href="https://github.com/JakeWharton/ActionBarSherlock/issues?milestone=4&state=open">under milestone 4.0.0</a> on the GitHub issue tracker. As always, code contributions are welcomed and encouraged.</p>
<p>There is no timeline yet for the final release. There will be one or two release candidates before which is when I will be working with a few devs on real implementations to determine any problems that exist. If everything goes smoothly the final release will not be far behind that.</p>
<p>Thank you everyone for your support thus far. Happy new year to all.</p>
ActionBarSherlock - A Love Story (Part 2)2011-12-19T00:00:00+00:00https://jakewharton.com/actionbarsherlock-a-love-story-part-2<p>Despite this, however, most users probably have never used the first or second versions, let alone the “lost” third version which was scrapped just hours from release. In future posts we’ll look forward to where version four will take us. For this post, however, we’ll be taking a quick look back at the origins of the library.</p>
<p>Like most libraries, ActionBarSherlock was birthed out of personal necessity. With my recent migration of the majority of our servers at work to VMWare and the vSphere platform in January 2011, I wanted an app which allowed me to view essential VM information and perform quick vMotions from my phone. At this time there was only two apps of exceptionally inferior quality on the Android Market which offered such functionality—neither being open source. It’s easy to guess what happened next: I began down the path of writing my own.</p>
<p>Writing an app which interfaces with vCenter Server (essentially vSphere’s coordination hub) is no trivial task. I spent a month porting the Java SDK to run on Android. During this process of ripping and replacing I ended up writing my own SOAP client which married aggressive caching with lazy loading objects. During this I discovered a lot of fun little-known facts about Android and Dalvik (Did you know that when reflecting on a class’ properties Android will return them in alphabetical order while desktop Java returns them in declaration order?). After about one month I had a mostly-working API wrapper which led to the next logical step, creating the application shell.</p>
<p>At that time, and much to my amusement today, I thought <a href="http://greendroid.cyrilmottier.com/">GreenDroid</a> to be the end-all, be-all library for implementing the common user interface patterns easily. If you’re following the timeline in your head, you might have already guessed that the Honeycomb SDK had landed this very same week and I chose to be forward thinking and support both phones and tablets with a single APK. This would, however, necessitate a bit of work since we were pre-Android Compatibility Library. I set off to adopt a proxy the action bar API of both GreenDroid and Honeycomb in a single custom API.</p>
<p>The <a href="https://github.com/JakeWharton/ActionBarSherlock/tree/1.0.0">first version</a> was completed in one day. It proxied only the methods which I needed and required you implement two static inner-classes to handle the pre- and post-Honeycomb configurations. I <a href="https://github.com/cyrilmottier/GreenDroid/issues/18">discovered</a> that both GreenDroid and Android used the <code class="highlighter-rouge">getActionBar()</code> method name which required a last-minute switch to <a href="https://github.com/hameno/GreenDroid">Hameno’s fork</a> where it was changed to <code class="highlighter-rouge">getGDActionBar()</code>. I had also discovered the compatibility library had launched and hastily slapped a mention of compatibility in the README file.</p>
<p>If you followed the link above you’ll notice that the entire library was <a href="https://github.com/JakeWharton/ActionBarSherlock/blob/1.0.0/ActionBarSherlock.java#files">a single class file</a> and <a href="https://github.com/JakeWharton/ActionBarSherlock/blob/1.0.0/sample/src/com/jakewharton/android/actionbarsherlock/sample/HelloActionBarActivity.java#files">a rudimentary sample</a>—a far cry from what it’s grown into today. Not very impressive, comprehensive, or even useful. I still dealt with menu inflation and action item creation manually!</p>
<p>The <a href="https://github.com/JakeWharton/ActionBarSherlock/tree/2.0.0">second version</a> was released the next day, a complete rewrite. I came to realize that GreenDroid just wasn’t going to cut it despite its numerous beautiful widgets and added better support for implementing your own “pre-Honeycomb” handler. <a href="https://github.com/johannilsson/android-actionbar">Android-ActionBar</a> support was added, my new library of choice for the vSphere client app (you almost forgot, didn’t you).</p>
<p>Over the next few weeks I managed to get simple screens of the application working such as listing VMs, viewing their information, and browsing things like the datacenter objects and datastores. As more screens were introduced, a more dynamic action bar was required and as such I implemented more and more of the native ActionBar API. Two weeks after 2.0.0, I released <a href="https://github.com/JakeWharton/ActionBarSherlock/blob/2.1.0/CHANGELOG.md#readme">version 2.1.0</a> which represented the first step towards where the library exists today. The compatibility was a required dependency, Maven was adopted as the build and release system, and APIs such as list navigation, menu inflation, and Fragment support was added.</p>
<p>At this point there were a handful of users who knew of the library and likely even less using it, progress on the VM app had stalled because of issues in getting my custom SOAP client working properly, and I had become aware of just how limiting my custom API really was. The following code snippet was taken from one of the samples of version 2.1.1:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">ActionBarSherlock</span><span class="o">.</span><span class="na">from</span><span class="o">(</span><span class="k">this</span><span class="o">)</span>
<span class="o">.</span><span class="na">with</span><span class="o">(</span><span class="n">savedInstanceState</span><span class="o">)</span>
<span class="o">.</span><span class="na">layout</span><span class="o">(</span><span class="no">R</span><span class="o">.</span><span class="na">layout</span><span class="o">.</span><span class="na">activity_hello</span><span class="o">)</span>
<span class="o">.</span><span class="na">menu</span><span class="o">(</span><span class="no">R</span><span class="o">.</span><span class="na">menu</span><span class="o">.</span><span class="na">hello</span><span class="o">)</span>
<span class="o">.</span><span class="na">homeAsUp</span><span class="o">(</span><span class="kc">true</span><span class="o">)</span>
<span class="o">.</span><span class="na">title</span><span class="o">(</span><span class="no">R</span><span class="o">.</span><span class="na">string</span><span class="o">.</span><span class="na">hello</span><span class="o">)</span>
<span class="o">.</span><span class="na">handleCustom</span><span class="o">(</span><span class="nc">ActionBarForAndroidActionBar</span><span class="o">.</span><span class="na">Handler</span><span class="o">.</span><span class="na">class</span><span class="o">)</span>
<span class="o">.</span><span class="na">attach</span><span class="o">();</span>
</code></pre></div></div>
<p>While there was nothing fundamentally wrong with this API in the context of my app, requests were coming in for the support of other action bar features. Constantly implementing methods for every item was clearly going to create a mess of a library. I chose once again to embark on a rewrite to afford a more flexible model.</p>
<p>Over the next month I reworked the entire library to be based on interfaces which provided a single action bar feature. This way, you could drop-in whatever third-party action bar library you wanted and only implement the feature interfaces in its handler that the library supported. On May 12th, 2011 <a href="https://github.com/JakeWharton/ActionBarSherlock/tree/83283d9f3eecc79762060f3dbcdbf09975c890f2">the code</a> was feature complete and ready for a 3.0.0 release. This tree, <code class="highlighter-rouge">83283d9f</code>, is the “lost” 3.0.</p>
<p>At some point during that evening—and I have no recolection of the exact moment—I had an epiphany.</p>
<blockquote>
<p>If I am providing an API for various action bar methods through a custom class, why am I not just providing the full API through a <code class="highlighter-rouge">getSupportActionBar()</code> method exactly like the compatibility library that I’m already dependent on?</p>
</blockquote>
<p>It turns out that adapting my code to provide the full action bar API was the easy part and didn’t take long. The majority of the next month was spent working with Johan Nilsson to <a href="https://github.com/johannilsson/android-actionbar/pull/25">expand Android-ActionBar’s feature set</a> to very nearly match that of Honeycomb.</p>
<p>Finally, on June 5th, 2011, version 3.0.0 was released which fully internalized the Android-ActionBar sources for a seamless mirroring of the native API on Android 1.6 and newer.</p>
<p>Releases on the 3.x branch <a href="https://github.com/JakeWharton/ActionBarSherlock/tags">came steadily</a> over the next few months culminating in <a href="https://twitter.com/#!/JakeWharton/status/148585409708965889">the release of 3.5.0</a> last night. If you’re reading this, you likely came aboard the ActionBarSherlock ship sometime during this time and probably reasonably familiar with how it operates. Through the support and contributions of the community it’s become quite a useful library—even solving problems such as supporting MapViews in fragments and preference activities.</p>
<p>I want to especially thank <a href="https://github.com/cyrilmottier">Cyril Mottier</a> of GreenDroid fame and <a href="https://github.com/johannilsson">Johan Nilsson</a> of Android-ActionBar fame. Despite now being my competition (not really), without their efforts the library would likely not exist. Thank you to all of the users who sent in pull requests. Thank you <a href="https://github.com/chrisbanes">Chris Banes</a> for fleshing out device support and providing tiny bug fixes and enhancements.</p>
<p>Thank you to all of the implementations. <a href="https://market.android.com/details?id=com.battlelancer.seriesguide">SeriesGuide</a>, <a href="https://market.android.com/details?id=com.ratebeer.android">RateBeer</a>, <a href="https://market.android.com/details?id=uk.co.senab.blueNotifyFree">FriendCaster</a>, <a href="https://market.android.com/details?id=com.minus.android">Minus</a>, <a href="https://market.android.com/details?id=com.strategiesinsoftware.erg">Cargo Decoder</a>, <a href="https://market.android.com/details?id=com.abcOrganizer">Folder Organizer</a>, <a href="https://market.android.com/details?id=com.mugitek.analytics">mAnalytics</a>, <a href="https://market.android.com/details?id=com.florianmski.tracktoid">Traktoid</a>, <a href="https://market.android.com/details?id=com.agilevent.crossfittravel">CrossFit Travel</a>, <a href="https://market.android.com/details?id=com.bubblesoft.android.bubbleupnp">BubbleUPnP</a>, <a href="https://market.android.com/details?id=com.mikedg.android.bar.lite">Bird Bar</a>, and the many, many more! <em>(I’m working on a webapp to organize all of the implementations)</em></p>
<p>Now, for the second time, did you remember this blog post was about a vSphere application? The explosion of interest in ActionBarSherlock as well some of my other projects overwhelmed my free time and coupled with my unstable SOAP client I never was able to make any more progress. In fact, I have never written an application using any of my own libraries. <em>…yet!</em></p>
<p>This week we’re not pouring one out, but rather I raise my glass to all of you, the ActionBarSherlock community. Cheers! See you at 4.0.</p>
<p><em>In the next installment of this series I will begin to talk about where version 4.0 will take us and how it is a return to its roots.</em></p>
ActionBarSherlock - A Love Story (Part 1)2011-12-01T00:00:00+00:00https://jakewharton.com/actionbarsherlock-a-love-story-part-1<p>The library has been out for nearly 9 months already and has <a href="https://github.com/JakeWharton/ActionBarSherlock/tags">seen 22 releases</a> across 3 major versions. Fast approaching is the next major release, version 4.0. This will bring the full functionaity of Ice Cream Sandwich’s action bar to all relevant APIs and will be what I consider the first true release. I’m going to be writing a series of posts which talks about some various development decisions and the reasoning behind them as well as a bit of the history of the library. The series will culminate in the formal announcement and release of v4.0 some time in the near future. So strap in and hold on, with this first post I’m just going to tear off the band-aid…</p>
<p>I am officially dropping 1.6 support from ActionBarSherlock 4.0. This post originally was going to be <a href="https://twitter.com/JakeWharton/status/142402144874663936">a call for arguments</a> against this, but with the release of the <a href="http://developer.android.com/resources/dashboard/platform-versions.html">latest platform distributions</a> I’m rounding 1.6’s share down to 1% which was my mental event horizon for its support.</p>
<p>This will likely upset a few people. I know of a lot of apps leveraging my library that still support 1.6. I even have <a href="https://market.android.com/details?id=com.strategiesinsoftware.erg">an implementer who supports 1.5</a> because he has an extremely niche market with a bunch of users who own the Motorola i1!</p>
<p>I’ve said time and time again to developers that as long as the compatibility library supported 1.6 then so would I. For ActionBarSherlock 3.x this was an important thing because the library was so tightly integrated with the compat lib that it actually became a near drop-in replacement for it since its classes were included.</p>
<p><strong>This will all change with ABS 4.0.</strong> I have chosen to take the library in a completely separate direction. While the next post will talk about the trials and tribulations of dealing with having a library which extends (and effectively replaces) the official compat lib, all we need to know for now is that the core library will now have zero dependencies.</p>
<p>Since the library is no longer immediately dependent on the compatibility library my justification for forcing support for Android 1.6 is no longer there. Now this is not to say that ActionBarSherlock 4.0 won’t be supporting tight integration with the compat lib—it most certainly will. I have instead simplified the main component of the library, the action bar, to stand alone.</p>
<p>As such, we can now focus solely on the betterment of the functionality that we’ve all come for. And with that comes waving a fond farewell to the version which I consider to be Android’s true “1.0”, API level 4.</p>
<p>To those that support Android 1.6 I commend you. It is in all sense of the word a bastard version. Development on Android was exploding and 1.5’s shortcomings were vastly documented (I’ll spare the links here) so 1.6 brought a bit of fresh air. In my opinion it was the first version of the platform that had the future in mind. Unfortunately for it, so were the next two API levels, the now defunct 2.0 and 2.0.1. Stability finally started to arrive in API 7, Android 2.1, which is to be the new minimum target of the library. This period of unrest saw the death of my primary argument against support 1.6, its classloader.</p>
<p>Android’s 1.6 classloader is an over-eager know-it-all. It seeks out and checks every method call present in your loaded (key word!) classes. This means that even if you’ve blocked out a section in a check against <code class="highlighter-rouge">Build.VERSION.SDK_INT</code>, Android 1.6 will check every method inside. This can be easily remedied by surrounding these calls in <a href="https://github.com/JakeWharton/ActionBarSherlock/blob/b0043b245eac671646b019dbbff55b2a4ec278c6/library/src/com/actionbarsherlock/internal/view/menu/ActionMenuPresenter.java#L121-126">concise static inner-classes</a> which is annoying, but doable. Problems arise when you <a href="https://github.com/JakeWharton/ActionBarSherlock/blob/da0bfadd1d546f97b92d9a93d028d2ac0113b49f/library/src/android/support/v4/app/FragmentActivity.java#L1010-1013">need to call the superclasses version of a method you have implicitly overriden</a>. You’re out of luck.</p>
<p>It turns out that the latter comes into play a lot in the ICS version of the action bar. Things such as accessibility and configuration changes on views are unable to call up to their superclass implementations. While probably not a show-stopper, couple this with constantly having to battle the former situation and it becomes a seemingly neverending battle of abstraction to these static classes.</p>
<p>My choice to drop Android 1.6 mostly comes out of wanting to be able to more rapidly develop and ship the library. I charge you, the reader (and hopefully ActionBarSherlock user), with the responsibility of determining whether maintaining support for 1.6 is feasible and also implementing it if you deem it so.</p>
<p>I’m sure some users will be still be angered by this decision despite the above explanation. Having developed, supported, and maintained this library for 9 months completely in my spare time and never charging a penny, I say, “<a href="http://www.youtube.com/watch?v=OaiSHcHM0PA">Show me the money.</a>” If you want to sponsor development of 1.6 support I will make every effort possible towards the effort. However, I’d rather you forked it, did it yourself, and sent a pull request… or better yet, just let it die.</p>
<p>Pour one out for Android 1.6 and all the users of it.</p>
<p><em>In the next installment of this series I will be talking about the history of ActionBarSherlock and how version 4.0 is a return to its roots.</em></p>