FWIW, I tried to cut down the number of files dagger checks for annotations this morning, and the build time managed to drop to less than 5 minutes.
CPU is still always at 100%
First 2 to 3 Incremental builds is usually slow as well, especially if I touched any of the annotated class files. But everything soon speeds up afterwards.