CODEWITHSUNDEEP
numpyspacyspacy-transformersdata-scrubbing
Dan
Dan

Reputation: 1

Spacy using all available CPU when running via Docker

I'm using scrubadub_spacy to clean text documents. When processing a document in a Docker environment built on amazonlinux, it maxes out the 8 cores on my MacBook and in Amazon Fargate. I'm confused why this is happening. The problem sounds very similar to the one solved here, but it sounds like Spacy now uses Blis, which is single-threaded, rather than BLAS. Also confusing is that if I install scrubadub_spacy in a virtual environment on my MacBook, I see about 3x better performance than when it is running in the container. Would anyone be able to help me understand what is going on here?

I was expecting the inference code to run on a single core and for performance to be similar when running in a container as when running outside of one.

Upvotes: 0

Views: 45

Answers (0)

Related Questions