The following content is translated from the original
In this article, we introduce the first multilingual sign language dataset called Prompt2Sign, which is based on public sign language data that includes American Sign Language (ASL) and seven other languages.
The dataset converts large amounts of video into a simplified, model-friendly format and is optimized for training translation models such as seq2seq and text2text. Based on this new dataset, SignLLM is proposed, the first multilingual sign language generation (SLP) model that includes two novel multilingual SLP patterns that allow sign language gestures to be generated based on input text or prompts.
Both models can use new loss and reinforcement learning-based modules to speed up training by enhancing the model’s ability to autonomously sample high-quality data. Benchmark results from SignLLM are demonstrated, which shows that our model achieves state-of-the-art performance on SLP tasks across eight sign languages.
Data sets and main methods
(Left) Overview of the structure and form of the PROMPT2SIGN dataset. (C) The interaction principle between Text2LangGloss and MLSF, and the calculation method of reinforcement learning. (Right) The output of SIGNLLM can be converted to most pose representation formats, which can then be rendered to a realistic human appearance through a style-transferred/specially fine-tuned generated model.
other methods
At work, the Text2Gloss framework was improved by incorporating a single tag that generates Gloss with necessary linguistic attributes, while also representing profound features through variables V and Xu in the neural network.
In addition, five key elements are introduced-users, agents, environments, iterative update processes, and PLCs-that together outline a reinforcement learning process tailored to sequence prediction.
If you want to learn more, you can click on the link below the video.
Thank you for watching this video. If you like it, please subscribe and like it. thank
Original text:https://signllm.github.io/
Paper：https://arxiv.org/abs/2405.10718
Oil tubing: