How You.com Applies Spark + Hugging Face Transformers for Search at Scale
Ғылым және технология
A multi-group, Sept 13 2022 NLP meetup presentation, "How You.com Applies Spark + Hugging Face Transformers for Search at Scale," with Sam Bean and Saahil Jain of You.com, moderated by Seth Grimes.
For the New York Natural Language Processing meetup, visit www.meetup.com/ny-nlp/.
Пікірлер: 4
Great meetup, thx. Loved the part about Spark cpu transformer embedding calculation for semantic indexing.
This is so brilliant. It's so hard to find these real-life learnings of how to deliver world-class ML systems in production. Especially the interaction between spark and transformers.
@jasonmackay891
Жыл бұрын
100%
Question, why not just use Apache Hive as a Catalog meta database, like the one found in AWS Glue. No need to move the data, since the catalog mdb points to the data. Moving data is costly, just saying. 18:44 sounds like we are in a pickle :)