If you use “Textarea” editor in Front End PM PRO > Settings > General > Editor Type then i can write some code for you to achieve this. Otherwise it is not possible.
Let me know if you need code.

Add following code in your theme’s (child theme’s if you are using) functions.php

add_action( 'fep_after_form_fields', function( $where, $errors, $fields ){
	if( in_array( $where, [ 'newmessage', 'reply' ] ) ) {
		?>
		<script type="text/javascript">
		document.addEventListener("keyup", function(event) {
			// Number 13 is the "Enter" key on the keyboard
			if (event.keyCode === 13) {
				// Cancel the default action, if needed
				event.preventDefault();
				document.querySelector('.fep-form').submit();
			}
		}); 
		</script>
		<?php
	}
}, 10, 3);