The structure of aneural network works through a training process, known as deep learning, a subfield of machine learning. This is artificial...
7 min read
20/04/2023
On March 22, 2023, we attended SlatorCon Remote, an online event where language industry experts and leaders came together to discuss this fascinating industry that is in constant growth. Some of the hot topics are new emerging markets and how productivity can be optimized through innovative language technologies, from machine translation and NLP to workflow automation.
Our Head of Machine Learning, (now CTO) Konstantinos Chatzitheodorou, gave a presentation in which he reflected on the tools that are making their mark with artificial intelligence. He examined the current status of large language models (LLMs), such as GPT-4, as well as their potential in the future of natural language processing tasks such as machine translation, summarization, and anonymization, and which may be its impact on the language industry.
Interesting news:
Our ECO Platform Integrates ChatGPT With Personal Data Privacy Features
The presentation and the most relevant questions:
Could you explain what large language model (LLM) means?
These models are trained on massive amounts of text data, allowing them to generate highly sophisticated and contextually relevant responses to human prompts. They can generate human-like text by predicting the next word or phrase based on a given context. This is old technology that has already been used in various applications, such as chatbots, customer service, content creation, and even translation. The difference now is that of context. Whilst statistical models are focused on 3, 5, or 7 n-grams, and neural machine translation on a sentence, GPT-4 can take up to 1500 words (or tokens) of context, thus providing a text that really flows.
Add autoregression, a lot of labeled data, and reinforcement training, and there you have it. Although OpenAI claims to have used only 30,000 human queries for its reinforcement!
More information:
ChatGPT is a generative language transformer developed by OpenAI, which is theoretically able to answer any question posed to it while maintaining a high degree of language accuracy.
Here is an example of how an LLM works:
A user prompts the question "What is the capital of France?" The LLM first analyzes the individual words in the sentence and determines their meaning and relationship. So, it recognizes that "capital" refers to a city, "France" refers to a country, and "what" is a request for information. Based on its understanding of language, it knows that the answer is "Paris" and it generates the response.
Should you inquire about its capabilities, it will provide the following response:
- Text Generation: Generates coherent and grammatically correct text
- Text Completion: Predicts sentence or paragraph completion
- Question Answering: Generates relevant responses to the question.
- Text Summarization: Generates summaries of longer texts.
- Sentiment Analysis: Classifies text sentiment (positive, negative, or neutral).
- Language Translation: Translates text between languages.
ChatGPT has quickly gained popularity, with over 100 million users utilizing the platform in under two months, causing widespread disruption in the technology industry. It can even be helpful for language learning, paraphrasing, etc.
Can LLMs replace humans?
Generative LLMs such as GPT-3 and 4 can generate human-like text. However, it is important to highlight that these models are not designed to replace humans, but rather to assist them in various tasks.
While GPT models can perform certain language-related tasks with a high level of accuracy, they lack human reasoning, empathy, and critical thinking abilities.
Additionally, these models are limited to the data that they have been trained on, which can result in bias, inaccuracies, or inconsistencies.
To sum it up, they are indeed impressive and useful tools, but they are not intended to replace humans. Instead, they can assist humans in performing tasks quickly and more efficiently, which can significantly enhance productivity and creativity.
More information:
How the New ChatGPT Will Send Ripples of Change Through the World as We Know It
Are there any restrictions to ChatGPT/LLMs?
Indeed, there are some limitations to LLMs. One major concern is where our data is stored and whether it will be utilized for developing new models. Regarding the question of where data is saved, OpenAI does not disclose the specific location of the servers. However, they do state that they use industry-standard security measures to protect the data and comply with relevant privacy laws.
It may be challenging for them to comprehend highly technical or specialized language that requires specific domain knowledge.
Additionally, they may not be suitable for sensitive or confidential information, such as legal or medical documents. Handling sensitive data is a critical concern when using ChatGPT or any other LLM. To address these concerns, various techniques can be used to ensure that sensitive data is handled appropriately. One approach is to use a data anonymization service (Pangeanic’s anonymization service can be used) to remove identifying information from the data before the model processes it. For example, names, addresses, and other identifying information can be replaced with anonymous placeholders, such as "Person A" or "Location B."
As I mentioned earlier, at Pangeanic, we have integrated ChatGPT into our workflow to enable our clients to generate text summaries. The process is simple and seamless: clients can upload their documents through our ECO platform and receive summaries. However, prior to submitting the information to ChatGPT, the documents are anonymized for security and privacy purposes.
In what ways can LLMs benefit the language industry?
Language service providers have already begun to utilize ChatGPT and other LLMs in their workflows. Since the first day of OpenAI's API release, they have integrated it for various use cases, such as generating product descriptions for e-commerce from templates, and translation. In the future, there are plans to expand the use of LLMs for other applications, such as translation quality estimation, file format conversion, and automatic post-editing.
Professionals recognize and appreciate the potential of LLMs, but they are generally skeptical about fully adopting them. Although they wish to see more examples, they also want to play a part in the implementation and application of LLMs within their organizations.
More information:
LSPs and professionals have long been the primary resource for language-related needs, but with the emergence of LLMs like ChatGPT, there are now more options available. The use of LLMs is likely to increase in the future and will play an important role in the language industry.
In addition, the industry has the potential to collaborate with and support other sectors by utilizing LLMs. This creates new opportunities for all those involved, including language professionals, developers, and business leaders. The development and integration of LLMs provide exciting possibilities for language-related services and can help to meet the ever-growing demand for multilingual communication.
Developing new models, competition, and language bias
The emergence of new models in the field of LLMs is almost a daily occurrence, causing constant significant changes in the landscape. In your opinion, what does the future hold for LLMs?
Indeed, new and innovative models are being announced daily by big organizations, research groups, and so on, as developments in the field of language models continue to progress at a rapid pace.
Over the past few days, LLMs will be integrated with Google Workspace, and Microsoft released an AI co-pilot for 365. It is highly likely that other widely used tools will follow suit in the coming days.
However, as is easily understood, the popular LLMs are predominantly trained on American English content, resulting in a US-centric bias in language and culture. Therefore, Europe and other nations should go for alternatives that will be competitive, will incorporate cultural diversity, and will preserve their languages and knowledge.
Furthermore, it is crucial to refrain from uploading essential data to services based in the United States. This strategy would enable developers to create models that are more culturally aware and less biased towards a specific region, promoting diversity and inclusivity in AI.
What does the future hold for LLMs?
The future of LLMs is promising and they are expected to play a significant role in various fields, including the language industry.
Please note that almost every day a new model/approach is released. For instance, just last week:
-
Stanford University released Alpaca 7B
-
OpenAI released GPT-4
-
Anthropic released Claude
-
Google announced the PaLM API and MakerSuite
-
Google added Generative AI to workspaces
-
Midjourney released V5
-
AssemblyAI released conformer1 (leverages Transformer and Convolutional layers for speech recognition)
-
There are previous models such as Bloom which are free even for commercial use.
Of course, we should highlight Meta’s LLaMA, a foundational, 65-billion-parameter LLM, as well as Google’s LLM which supports automatic speech recognition and more than 1,000 of the world’s most-spoken languages.
Another potential future for LLMs is the development of more specialized models for specific industries or domains. For example, a model designed specifically for the legal industry could be trained on legal terminology and jargon, making it better equipped to handle legal documents. This would help address some of the limitations of LLMs when it comes to domain-specific language.
Multimodality is also an area of innovation that is becoming increasingly important in the development of LLMs. Multimodal models are able to understand and process multiple forms of input, such as text, images, video, and audio, which can help to improve the accuracy and performance of natural language processing tasks.
One of the key advantages of multimodal LLMs is that they can help bridge the gap between different means of communication, allowing us to generate more coherent and comprehensive responses that incorporate multiple forms of input.
This could have a wide range of applications, from improving the accuracy of image and speech recognition systems to enhancing the capabilities of chatbots and virtual assistants.
Pricing is an important factor. Multimodal LLMs can be expensive, but they are often worth the investment. The development and training of these models require substantial resources, including computing power, data sets, and skilled personnel. However, the benefits of using LLMs, such as their ability to process vast amounts of data quickly and accurately, can far outweigh the costs. These models can provide businesses with valuable insights and enable them to make data-driven decisions. Moreover, LLMs can be used to automate many tasks, freeing up human resources for more specialized and creative work. Overall, while LLMs may come with a high price tag, their potential benefits make them a worthwhile investment for many organizations.
The future of LLMs could be improved through integration with other technologies. Augmented reality and virtual reality, for example, could be used to create immersive language learning experiences. Additionally, mobile devices are a natural fit for LLMs and will likely play a significant role in their integration. Multilingualism and integration with different industries beyond the language industry, such as healthcare, finance, and law, are also important areas for researchers to focus on.
By exploring these avenues, LLMs can expand their applications and continue to provide value in a wide range of contexts.
While ChatGPT is currently known for its primary use in translation, quality assurance, or summarization, I believe its capabilities have the potential to extend beyond this.
My understanding is that ChatGPT's technology is versatile and can be applied in various areas within the language industry. As such, I hold a firm belief that ChatGPT's potential applications could encompass tasks such as forecasting and pricing, among others.
More information:
Its ability to process large amounts of data and interpret complex patterns could be a game-changer for businesses seeking to optimize their operations and gain a competitive edge. In my view, ChatGPT's potential to revolutionize industry practices is significant and warrants further exploration.
