• 0 Posts
  • 8 Comments
Joined 11 months ago
cake
Cake day: December 18th, 2023

help-circle






  • It can and it will. That is one of the uses of “NPUs” I’m most excited about.

    Basically you can run an (potentially open-source) small LLM on the phone using whatever context the keyboard has access to (at a minumim, what you’ve typed so far) and have the keyboard generate the next token(s).

    Since this is comptationally intensive the model has to be small and you need dedicated hardware to optimize it, otherwise you would need a 500W GPU like the big players. You can do it for 0.5W locally. Of course, adjust your expectations accordingly.

    I don’t know any project doing it right now, but I imagine that Microsoft will integrate in SwiftKey soon, with open source projects to follow.