Release 0.2.0

.gitignore

@@ -175,3 +175,13 @@ cython_debug/
 
 # PyPI configuration file
 .pypirc
+
+# Helm
+**/charts/*.tgz
+
+# Others
+.trash
+docs
+
+# HuRI client outputs
+*.wav

Makefile

@@ -0,0 +1,10 @@
+lint:
+	black .
+	isort .
+	flake8 .
+	mypy . --check-untyped-defs
+
+test:
+	pytest
+
+check: lint test

README.md

@@ -1,3 +1,88 @@
-# speech-to-speech
+# HuRI
 
-1. python3.12
+## Presentation
+
+HuRI is an open-source research project focused on conversational AI for humanoid robots and virtual avatars.
+HuRI provides a modular architecture that allows developers to design, implement, and run AI modules within customizable conversational pipelines defined by the user.
+HuRI is launched as a multi-client server, to handle multiple client (robots) conversational requests.
+The framework supports the implementation and integration of multiple AI modules, including:
+Speech-to-Text (STT) and Text-to-Speech (TTS), Retrieval-Augmented Generation (RAG), Emotional analysis (EMO), Motion and gesture generation (MOV)
+
+## Getting Started
+
+### Prerequisites
+
+- python 3.11.14
+  ```sh
+  sudo apt install python3.11
+  ```
+- pip
+  ```sh
+  sudo apt install python3-pip
+  ```
+
+### Installation
+
+1. Clone the repo
+   ```sh
+   git clone https://github.com/Sentience-Robotics/HuRI.git
+   ```
+2. Install pip packages
+   ```sh
+   pip install -r requirements.txt
+   ```
+
+## Usage
+
+#### Launch HuRI server:
+
+```sh
+serve run [config_file_path]
+```
+
+We use ray serve config file, doc [here](https://docs.ray.io/en/latest/serve/configure-serve-deployment.html).
+
+You can also launch HuRI without config file:
+
+```sh
+python -m src.launch_huri
+```
+
+#### Launch Client:
+
+```sh
+python -m src.client --config [client_config_file_path]
+```
+
+We have custom yaml file to define modules to use and how they are initialized, template [here](config/client_template.yaml).
+
+### Folder/Module structure
+
+- #### root
+
+Entrypoints:
+
+HuRI: app.py & launch_huri.py \
+Client: client.py
+
+- #### Core
+
+HuRI's Core classes:
+
+client & client_senders \
+events \
+huri \
+module \
+session
+
+- #### Modules
+
+Modules implementation:
+
+rag: rag's implementation \
+speech_to_text: speech to text implementation, including MIC (vad), STT (speech to text) and TAG (text aggregator) \
+utils: utility modules, like Sender (send event to Clients)
+
+## Developper Documentation
+
+HuRI's complete documentation is available [here](https://docs.sentience-robotics.fr/share/p1x9ikjkhf/p/hu-ri-documentation-f4amcndYQg).

btp.md

@@ -1,4 +1,4 @@
-### **BETA TEST PLAN – TEMPLATE**
+### **BETA TEST PLAN**
 
 ## **1. Project context**
 HuRI is a research-driven platform that has the objective to act as a universal middleware for speech and movement for any various physical or digital human embodiments. The focus on our part is to have embodiments as human as possible, regarding the emotion aspect, and try to go beyond the uncanny valley. The goal is to create a modular ecosystem where any combination of AI can control any robot or any virtual avatar. The project focuses on a highly stable architecture, scalable cloud deployment, and innovative emotional expression through motion and voice.
@@ -7,55 +7,60 @@ HuRI is a research-driven platform that has the objective to act as a universal
 
 | **Role Name** | **Description** |
 |---------------|-----------------|
-| Library Maintainer | Manages the core Python package, handles versioning, and ensures the stability of the base classes and API. |
 | Module Developer | Uses the library to build and package custom plugins or specialized modules for specific use cases. |
 | DevOps Engineer | Handles the containerization, orchestration, and distribution of the library across different network nodes or edge devices. |
-| Client | User of the library |
+| Client | Integrates the library into their own projects without concern for its internal implementation. |
 
 
 ## **3. Feature table**
 
 | **Feature ID** | **User role** | **Feature name** | **Short description** |
 |--------------|---------------|-------------------------|--------------------------------------|
-| F1 | Everyone | Create Module | The lib allows to create a module and use it. |
-| F2 | Everyone | Config file | Config file to launch different module combinations.  |
-| F3 | Everyone | Parallelism | HuRI can run on 1 to N machines, to split computation and balance payload. |
-| F4 | Everyone | Multi-client | HuRI can be used on 1 to N robots with separated discussions. |
-| F5 | Everyone | Module - MIC | User can talk to HuRI through microphone. |
-| F6 | Everyone | Module - SPK | HuRI can talk to User through speakers. |
-| F7 | Everyone | Module - STT | HuRI can transcribe speech into text. |
-| F8 | Everyone | Module - INP | User can chat to HuRI through terminal. |
-| F9 | Everyone | Module - OUT | HuRI can chat to User through terminal. |
-| F10 | Everyone | Module - MOD | HuRI has 3 modes: Discussion, Inserting context & Inserting information. User can switch between modes. |
-| F11 | Everyone | Module - TTS | HuRI can generate speech with text. Generate audio. |
-| F12 | Everyone | Module - MOV | HuRI can generate body movement by putting points in space |
-| F13 | Everyone | Module - RAG | HuRI can retrieve text from files, saved texts, old conversations, etc. |
-| F14 | Everyone | Module - LLM | HuRI can generate text from a given context |
-| F15 | Everyone | Module - TAN | User speech text will be analysed to understand their emotion. |
-| F16 | Everyone | Module - EIN | Analysed emotion is mixed with the context. |
-| F17 | Everyone | Module - AMM  | HuRI will store and manage an artificial memory |
+| F1 | Module Developer | Create a Module | Build and integrate modules into the library. |
+| F2 | Module Developer & DevOps Engineer | Configure HuRI | Define available modules and deployement settings for HuRI. |
+| F3 | Client | Configure Modules | Define and load different module combinations through a configuration file. |
+| F4 | Everyone | Run in Parallel | Distribute computation across 1 to N machines to balance the payload. |
+| F5 | Everyone | Handle Multi-client | Handle 1 to N simultaneous client instances with separated discussions. |
+| F6 | Everyone | Detect Voice Activity | Detect when the user is speaking. |
+| F7 | Everyone | Transcribe Speech |  Generate text from an audio speech. |
+| F8 | Everyone | Generate Speech | Generate speech audio from a given text input. |
+| F9 | Everyone | Generate Body Movement | Generate body movements by placing points in space. |
+| F10 | Everyone | Retrieve and Augment | Retrieve text from files, saved texts, or past conversations and generate an output. |
+| F11 | Everyone | Recognise Speech Emotion | Recognise emotions from user speech. |
+| F12 | Everyone | Link Emotion to Input | Link a transcript with its associated emotion. |
+| F13 | Everyone | Manage Artificial Memory | Store and manage an artificial memory for HuRI. |
+| F14 | Everyone | Send User Audio | Transmit user audio to the HuRI system. |
+| F15 | Everyone | Send User Text | Transmit user text to the HuRI system. |
+| F16 | Everyone | Receive Generated Audio | Deliver generated audio back to the user. |
+| F17 | Everyone | Receive Generated Text | Deliver generated text back to the user. |
+| F18 | Everyone | Store Vectorised Data | Store and retrieve vectorised data through a dedicated service. |
+| F19 | Everyone | Call LLM Service | Generate a response by calling an external LLM service. |
 
 ---
 
 ## **4. Success Criteria**
 
 | **Feature ID** | **Key success criteria** | **Indicator/metric** | **Result** |
 |--------------|---------------------------------------|-----------------------|----------------|
-| F1 | Creating various Modules. | 20 attempts -- expected 100% | Achieved (/20) |
-| F2 | Launching different scenarios with different config files. | 5 files -- expected 100% | Achieved (/5) |
-| F3 - F4 | Running 1 or several modules on 1 or several machines. | All modules running on 1 machine. All modules running on different machines. Several modules running on several machines. | Scenario achieved (/3) |
-| F5 | Using a microphone and being recorded. | 10 messages recorded over 3 different devices -- expected 100% | Files recorded (/30) |
-| F6 | Emitting sound through speakers. | 10 audio files played on 3 different devices -- expected 100% | Files heard (/30) |
-| F7 | Transcribing speech to text correctly. | 20 spoken phrases -- expected 80% accuracy | Phrases correct (/20) |
-| F8 | Sending text input via terminal. | 10 text inputs -- expected 100% | Inputs received from another module (/10) |
-| F9 | Receiving text output via terminal. | 10 text outputs from another module -- expected 100% | Outputs displayed (/10) |
-| F10 | Switching between the 3 modes (Discussion, Context, Info). | Switch 10 times -- expected 100% success | Switches successful (/10) |
-| F11 | Generating audio file from text input. | 10 text inputs -- expected 100% generation | Files generated (/10) |
-| F12 | Making human movement | human feeling* -- expected 60% | Human feedback in % |
-| F13 | Retrieving context from a saved file. | 10 queries on saved text -- expected 100% retrieval | Relevant text found (/10) |
-| F14 | Generating a response from a context. | 10 prompts -- expected 100% answer generation | Answers generated (/10) |
-| F15 | Understanding of emotions from the interlocutor. | 10 emotion texts to analyse -- expected 60% | Emotion analysed (/10) |
-| F16 | Understanding of the global emotion of the interlocutor | 10 emotional contexts to analyse -- expected 60% | 10 emotional contexts (/10) |
-| F17 | Saving of important information with an update that removes information when they are old / not relevant anymore / not remembered | 10 pieces of information to save and treat over time -- expected 60% | 10 prompts to save and treat (/10) |
+| F1 | Verify modules can be written and run inside HuRI. | 20 attempts -- expected 100% | Achieved (/20) |
+| F2 | Check that available modules and cluster settings correspond to the configuration. | 5 configurations -- expected 100% | Achieved (/5) |
+| F3 | Confirm that a specific module combination loads from a config file. | 5 files -- expected 100% | Achieved (/5) |
+| F4 | Test that HuRI runs across multiple machines with computation distributed. | 1 machine / N machines | Scenario achieved (/2) |
+| F5 | Verify multiple clients stay isolated with separate discussions. | 3 simultaneous instances -- expected 100% isolation | Instances isolated (/3) |
+| F6 | Check that speaking and silence are correctly detected. | 10 voice samples -- expected 100% | Detections successful (/10) |
+| F7 | Confirm spoken input is transcribed accurately. | 20 spoken phrases -- expected 80% accuracy | Phrases correct (/20) |
+| F8 | Verify a speech audio file with human characteristic is generated from a text input accurately. | 10 text inputs -- expected 100% generation -- expected 60% human feeling* | Files generated (/10) -- Human feedback in % |
+| F9 | Assess whether body movements feel human-like. | 10 movement sequences -- expected 60% human feeling | Human feedback in % |
+| F10 | Check that relevant saved content is retrieved to build an answer. | 10 queries on saved text -- expected 100% retrieval | Relevant text found (/10) |
+| F11 | Verify the emotion in user speech is correctly identified. | 10 emotion samples -- expected 60% accuracy | Emotions recognised (/10) |
+| F12 | Confirm the detected emotion is associated with the transcript. | 10 emotional contexts -- expected 60% | Contexts linked (/10) |
+| F13 | Test that information is retained, updated, and forgotten over time. | 10 pieces of info over time -- expected 60% | Prompts saved and treated (/10) |
+| F14 | Verify user audio is transmitted correctly to the HuRI system. | 10 audio inputs -- expected 100% | Inputs received (/10) |
+| F15 | Confirm user text is transmitted correctly to the HuRI system. | 10 text inputs -- expected 100% | Inputs received (/10) |
+| F16 | Check that generated audio is correctly delivered back to the user. | 10 audio files -- expected 100% | Files heard (/10) |
+| F17 | Confirm generated text is correctly delivered back to the user. | 10 text outputs -- expected 100% | Outputs displayed (/10) |
+| F18 | Verify vectorised data is stored and retrieved accurately. | 10 operations -- expected 100% | Operations successful (/10) |
+| F19 | Test that a coherent response is returned from the external LLM service. | 10 prompts -- expected 100% generation | Answers generated (/10) |
+
 
 *Experiments will be conducted on several people and a Godspeed-based questionnaire will evaluate the human feeling

config/client_aux.yaml

@@ -0,0 +1,28 @@
+huri_url: ws://localhost:8000/session
+
+topic_list: [question]
+
+senders:
+  audio:
+    name: audio
+    args:
+      sample_rate: 16000
+      frame_duration: 0.030
+
+modules:
+  mic:
+    name: mic
+    args:
+      vad_agressiveness: 3
+      silence_duration: 1.5
+      block_duration: ${inputs.audio.args.frame_duration}
+    logging: INFO
+  stt:
+    name: stt
+    args:
+      language: "en"
+      block_duration: ${inputs.audio.args.frame_duration}
+    logging: INFO
+  tag:
+    name: tag
+    logging: INFO

config/client_aux2.yaml

@@ -0,0 +1,32 @@
+huri_url: ws://localhost:8000/session
+
+topic_list: [transcript, question, rag_response]
+
+senders:
+  audio:
+    name: audio
+    args:
+      sample_rate: 16000
+      frame_duration: 0.030
+
+modules:
+  mic:
+    name: mic
+    args:
+      vad_agressiveness: 3
+      silence_duration: 1.5
+      block_duration: ${senders.audio.args.frame_duration}
+  stt:
+    name: stt
+    args:
+      language: en
+      block_duration: ${senders.audio.args.frame_duration}
+    logging: INFO
+  tag:
+    name: tag
+    logging: INFO
+  rag:
+    name: rag
+    args:
+      language: en
+      tone: formal

config/client_auxio.yaml

@@ -0,0 +1,25 @@
+huri_url: ws://localhost:8000/session
+
+topic_list: [question]
+
+senders:
+  text:
+    name: text
+
+modules:
+  mic:
+    name: mic
+    args:
+      vad_agressiveness: 3
+      silence_duration: 1.5
+      block_duration: ${senders.audio.args.frame_duration}
+    logging: INFO
+  stt:
+    name: stt
+    args:
+      language: en
+      block_duration: ${senders.audio.args.frame_duration}
+    logging: INFO
+  tag:
+    name: tag
+    logging: INFO