Spark Nlp

Spark NLP is an open source natural language processing library built natively on Apache Spark, so text pipelines can scale across a cluster. It is released under the Apache 2.0 license, including its pretrained models and pipelines, and offers Python, Scala and Java support. It installs from PyPI, Conda or Maven. The library ships with a large set of ready to use features. Classic text processing includes tokenization, word segmentation, stop word removal, stemming, lemmatization, n grams, regex and text matching, chunking, date matching, part of speech tagging, sentence detection, dependency parsing and spell checking. It also provides sentiment detection, text classification, keyword extraction, language identification covering up to 375 languages, and table question answering. For deep learning it brings transformer models to production on the JVM and Python, including BERT, RoBERTa, DeBERTa, XLM RoBERTa, Longformer, T5, BART, MarianMT, GPT 2, Vision Transformer, Wav2Vec2 and OpenAI Whisper. Embedding options include BERT, E5, MPNet, Instructor, Universal Sentence Encoder and OpenAI embeddings. A Models Hub hosts the pretrained models and pipelines. Spark NLP is aimed at data engineers and machine learning teams that need to run NLP and large language model workloads in production on Spark. The site cites a 2021 survey ranking it as the most widely used NLP library in the enterprise. The library is free to use. The site does not publish any paid plans.