Use WinAPI to simulate pointermovements

[qdlmcfresh]

Add pen support for #17
Touch is not implemented yet and at the current stage i think it will only compile on windows.
Since the coordinates sent by the event are not padded to fit the actual monitorlayout on a multimonitor setup, the pointer will allways be on the leftmost monitor.

The library captrs supports taking screenshots of multiple monitors, so this could be used instead of autopilot.

To get the number of connected and active displays this can be used.

use winapi::um::winuser::*;
use winapi::um::wingdi::{DISPLAY_DEVICEA, DISPLAY_DEVICE_ACTIVE};
fn get_connected_monitors() -> i32 {
    let mut i = 0u32;
    let mut connected_monitors = 0;
    unsafe {
        let mut dsp_dev: DISPLAY_DEVICEA = std::mem::zeroed();
        dsp_dev.cb = std::mem::size_of::<DISPLAY_DEVICEA>() as u32;
        while EnumDisplayDevicesA(
            std::ptr::null(),
            i,
            std::ptr::addr_of_mut!(dsp_dev),
            EDD_GET_DEVICE_INTERFACE_NAME,
        ) != 0
        {
            if dsp_dev.StateFlags & DISPLAY_DEVICE_ACTIVE != 0 {
                connected_monitors += 1;
            }
            i += 1;
        }
    }
    connected_monitors as i32
}

[H-M-H]

The library captrs supports taking screenshots of multiple monitors, so this could be used instead of autopilot.

Hmm, how does captrs work, it looks like you can pass some kind of ID to new: https://docs.rs/captrs/0.3.1/captrs/struct.Capturer.html#method.new, how does it map to monitors?

Regarding your implementation, it's better not to modify src/input/autopilot_device.rs as it's still required for macOS but instead create a new file say src/input/winapi_device.rs that implements

Weylus/src/input/device.rs

Lines 10 to 16 in e23d3b9

	pub trait InputDevice {
	fn send_wheel_event(&mut self, event: &WheelEvent);
	fn send_pointer_event(&mut self, event: &PointerEvent);
	fn send_keyboard_event(&mut self, event: &KeyboardEvent);
	fn set_capturable(&mut self, capturable: Box<dyn Capturable>);
	fn device_type(&self) -> InputDeviceType;
	}

and then instantiate it here:

Weylus/src/websocket.rs

Lines 513 to 517 in e23d3b9

	#[cfg(not(target_os = "linux"))]
	if self.input_device.is_none() {
	self.input_device = Some(Box::new(
	crate::input::autopilot_device::AutoPilotDevice::new(capturable.clone()),
	));

[qdlmcfresh]

The library captrs supports taking screenshots of multiple monitors, so this could be used instead of autopilot.

Hmm, how does captrs work, it looks like you can pass some kind of ID to new: https://docs.rs/captrs/0.3.1/captrs/struct.Capturer.html#method.new, how does it map to monitors?

Yep the id is the monitor nr. 0 is the primary

Regarding your implementation, it's better not to modify src/input/autopilot_device.rs as it's still required for macOS but instead create a new file say src/input/winapi_device.rs that implements

Weylus/src/input/device.rs

Lines 10 to 16 in e23d3b9

	pub trait InputDevice {
	fn send_wheel_event(&mut self, event: &WheelEvent);
	fn send_pointer_event(&mut self, event: &PointerEvent);
	fn send_keyboard_event(&mut self, event: &KeyboardEvent);
	fn set_capturable(&mut self, capturable: Box<dyn Capturable>);
	fn device_type(&self) -> InputDeviceType;
	}

and then instantiate it here:

Weylus/src/websocket.rs

Lines 513 to 517 in e23d3b9

	#[cfg(not(target_os = "linux"))]
	if self.input_device.is_none() {
	self.input_device = Some(Box::new(
	crate::input::autopilot_device::AutoPilotDevice::new(capturable.clone()),
	));

yep will do that this was just for fast testing

[H-M-H]

Yep the id is the monitor nr. 0 is the primary

That's nice, implementing a capturable should be straight forward then. There is a Capturable trait:

Weylus/src/capturable/mod.rs

Lines 33 to 47 in e23d3b9

	pub trait Capturable: Send + BoxCloneCapturable {
	/// Name of the Capturable, for example the window title, if it is a window.
	fn name(&self) -> String;
	/// Return x, y, width, height of the Capturable as floats relative to the absolute size of the
	/// screen. For example x=0.5, y=0.0, width=0.5, height=1.0 means the right half of the screen.
	fn geometry_relative(&self) -> Result<(f64, f64, f64, f64), Box<dyn Error>>;
	/// Callback that is called right before input is simulated.
	/// Useful to focus the window on input.
	fn before_input(&mut self) -> Result<(), Box<dyn Error>>;
	/// Return a Recorder that can record the current capturable.
	fn recorder(&self, capture_cursor: bool) -> Result<Box<dyn Recorder>, Box<dyn Error>>;
	}

and a Recorder trait

Weylus/src/capturable/mod.rs

Lines 16 to 18 in e23d3b9

	pub trait Recorder {
	fn capture(&mut self) -> Result<crate::video::PixelProvider, Box<dyn Error>>;
	}

that need to be implemented. Have a look at https://github.com/H-M-H/Weylus/blob/master/src/capturable/autopilot.rs for the most basic Capturable + Recorder. You could add a new captrs capturable that stores its monitor id. As captrs returns a &[] to

#[repr(C)]
pub struct Bgr8 {
    pub b: u8,
    pub g: u8,
    pub r: u8,
    _pad: u8,
}

(using capture_store_frame and get_stored_frame) it can be safely transmuted to a &[u8] with BGRx layout, which is fortunately exactly what Weylus uses internally as well. Finally the capturables need to be instantiated here:

Weylus/src/capturable/mod.rs

Line 55 in e23d3b9

pub fn get_capturables(

.

yep will do that this was just for fast testing

Alright!

[qdlmcfresh]

My capturer implementation doesnt compile. Dont really know what to do about that. Its my first time really using rust ...

use crate::capturable::{Capturable, Recorder};
use captrs::Capturer;
use std::boxed::Box;
use std::error::Error;

#[derive(Clone)]
pub struct CaptrsCapturable {
    id: u8,
    capturer: Capturer,
}

impl CaptrsCapturable {
    pub fn new(id: u8) -> CaptrsCapturable {
        let capturer = Capturer::new(id).unwrap();
        CaptrsCapturable { id, capturer }
    }
}

impl Capturable for CaptrsCapturable {
    fn name(&self) -> String {
        "Desktop (captrs)".into()
    }
    fn geometry_relative(&self) -> Result<(f64, f64, f64, f64), Box<dyn Error>> {
        Ok((0.0, 0.0, 1.0, 1.0))
    }
    fn before_input(&mut self) -> Result<(), Box<dyn Error>> {
        Ok(())
    }
    fn recorder(&self, _capture_cursor: bool) -> Result<Box<dyn Recorder>, Box<dyn Error>> {
        Ok(Box::new(CaptrsRecorder::new()))
    }
}

pub struct CaptrsRecorder {
    img: Vec<u8>,
}

impl CaptrsRecorder {
    pub fn new() -> CaptrsRecorder {
        CaptrsRecorder { img: Vec::new() }
    }
}

impl Recorder for CaptrsRecorder {
    fn capture(&mut self) -> Result<crate::video::PixelProvider, Box<dyn Error>> {
        self.capturer.capture_store_frame();
        let (w, h) = self.capturer.geometry();
        self.img = self.capturer.get_stored_frame().unwrap();
        Ok(crate::video::PixelProvider::BGR0(w, h, &self.img))
    }
}

`NonNull<ID3D11Device>` cannot be sent between threads safely
within `CaptrsCapturable`, the trait `Send` is not implemented for `NonNull<ID3D11Device>`rustcE0277
main.rs(1, 1): required because it appears within the type `wio::com::ComPtr<ID3D11Device>`
main.rs(1, 1): required because it appears within the type `dxgcap::DuplicatedOutput`
main.rs(1, 1): required because it appears within the type `std::option::Option<dxgcap::DuplicatedOutput>`
main.rs(1, 1): required because it appears within the type `dxgcap::DXGIManager`
main.rs(1, 1): required because it appears within the type `Capturer`
captrs_capture.rs(7, 12): required because it appears within the type `CaptrsCapturable`
mod.rs(34, 23): required by a bound in `Capturable`
`NonNull<ID3D11DeviceContext>` cannot be sent between threads safely
within `CaptrsCapturable`, the trait `Send` is not implemented for `NonNull<ID3D11DeviceContext>`rustcE0277
main.rs(1, 1): required because it appears within the type `wio::com::ComPtr<ID3D11DeviceContext>`
main.rs(1, 1): required because it appears within the type `dxgcap::DuplicatedOutput`
main.rs(1, 1): required because it appears within the type `std::option::Option<dxgcap::DuplicatedOutput>`
main.rs(1, 1): required because it appears within the type `dxgcap::DXGIManager`
main.rs(1, 1): required because it appears within the type `Capturer`
captrs_capture.rs(7, 12): required because it appears within the type `CaptrsCapturable`
mod.rs(34, 23): required by a bound in `Capturable`
`NonNull<IDXGIOutput1>` cannot be sent between threads safely
within `CaptrsCapturable`, the trait `Send` is not implemented for `NonNull<IDXGIOutput1>`rustcE0277
main.rs(1, 1): required because it appears within the type `wio::com::ComPtr<IDXGIOutput1>`
main.rs(1, 1): required because it appears within the type `dxgcap::DuplicatedOutput`
main.rs(1, 1): required because it appears within the type `std::option::Option<dxgcap::DuplicatedOutput>`
main.rs(1, 1): required because it appears within the type `dxgcap::DXGIManager`
main.rs(1, 1): required because it appears within the type `Capturer`
captrs_capture.rs(7, 12): required because it appears within the type `CaptrsCapturable`
mod.rs(34, 23): required by a bound in `Capturable`
`NonNull<IDXGIOutputDuplication>` cannot be sent between threads safely
within `CaptrsCapturable`, the trait `Send` is not implemented for `NonNull<IDXGIOutputDuplication>`rustcE0277
main.rs(1, 1): required because it appears within the type `wio::com::ComPtr<IDXGIOutputDuplication>`
main.rs(1, 1): required because it appears within the type `dxgcap::DuplicatedOutput`
main.rs(1, 1): required because it appears within the type `std::option::Option<dxgcap::DuplicatedOutput>`
main.rs(1, 1): required because it appears within the type `dxgcap::DXGIManager`
main.rs(1, 1): required because it appears within the type `Capturer`
captrs_capture.rs(7, 12): required because it appears within the type `CaptrsCapturable`
mod.rs(34, 23): required by a bound in `Capturable`
weylus::capturable
pub trait Capturable
where
    Self: Send + BoxCloneCapturable,

[H-M-H]

My capturer implementation doesnt compile. Dont really know what to do about that. Its my first time really using rust ...

Don't worry :) The Capturable + Recorder traits are designed in such a way that the Capturable can be safely copied and shared among threads. So a Capturable should only store minimal meta information about what it is (in this case which display) and actually capturing the screen should be delegated to the Recorder which doesn't need to be cloneable or thread safe.

The concrete problem you have run into is that captrs Capturer is not thread safe. This means it "poisons" your CaptrsCapturable struct. The solution is to just store the display ID inside the Capturable and instantiate the Capturer only with the Recorder.

I hope this clears things up, gotta get some sleep now, so I will take some time to reply if something comes up.

[qdlmcfresh]

the last thing missing now is a way to get the screensize from a capturable to calculate the coordinates. I thought about adding a screen_size function to the capturable trait but that wont work since in my case the recorder holds the captrs variable that has the screensize.

[H-M-H]

Looks like captrs uses EnumOutputs instead of EnumDisplayDevicesA internally, which can be used to get the output geometry https://docs.microsoft.com/en-us/windows/win32/api/dxgi/nf-dxgi-idxgioutput-getdesc. If that does not work, a somewhat dirty solution is to just create a captrs capturer, query its geometry, save that and throw away the capturer immediately afterwards. Not exactly beautiful but sure better than nothing.

In the longrun I'd like to get rid of third party crates for screen capturing though as non of those (at least to my knowledge) provide a way to capture individual windows which is something I'd like to see supported on all platforms (actually only the Windows implementation is missing). But that's probably a significant amount of work and I understand if you are not up for it.

[qdlmcfresh]

Another thing to look at is the coordinate offset. Sadly captrs implements a function to get the offset for linux but not for windows. The best thing i could get is this via EnumDisplayMonitors and GetMonitorInfo:

Monitor: 0
rcMonitor.left -1920 rcMonitor.top 0 rcMonitor.right 0 rcMonitor.bottom 1080
1080x1920
rcWork.left -1920 rcWork.top 0 rcWork.right 0 rcWork.bottom 1040
1040x1920
dwFlags: 0 

Monitor: 1
rcMonitor.left 0 rcMonitor.top 0 rcMonitor.right 2560 rcMonitor.bottom 1440
1440x2560
rcWork.left 0 rcWork.top 0 rcWork.right 2560 rcWork.bottom 1400
1400x2560
dwFlags: 1 // This means its the primary monitor

Calculating the offsets is weird though since: rcMonitor.left * -1 + event.x results in a correct coordinate for "Monitor 1" but i dont know how that maps to more monitors with different layouts than left->right.
I dont think dxgcap has the offset information either because the structs returned by the directx calls seem very similar to the ones i get with GetMonitorInfo

I have a working setup for 1 monitor though :)

Btw this doesnt return the same order as the EnumMonitorDevices method. Main Display is 1 with EnumMonitorDevices and 0 with EnumDisplayDevicesA for whatever reason

Add pen support for #17 Touch is not implemented yet and at the current stage i think it will only compile on windows. Since the coordinates sent by the event are not padded to fit the actual monitorlayout on a multimonitor setup, the pointer will allways be on the leftmost monitor.

The library captrs supports taking screenshots of multiple monitors, so this could be used instead of autopilot.

To get the number of connected and active displays this can be used.

use winapi::um::winuser::*;
use winapi::um::wingdi::{DISPLAY_DEVICEA, DISPLAY_DEVICE_ACTIVE};
fn get_connected_monitors() -> i32 {
    let mut i = 0u32;
    let mut connected_monitors = 0;
    unsafe {
        let mut dsp_dev: DISPLAY_DEVICEA = std::mem::zeroed();
        dsp_dev.cb = std::mem::size_of::<DISPLAY_DEVICEA>() as u32;
        while EnumDisplayDevicesA(
            std::ptr::null(),
            i,
            std::ptr::addr_of_mut!(dsp_dev),
            EDD_GET_DEVICE_INTERFACE_NAME,
        ) != 0
        {
            if dsp_dev.StateFlags & DISPLAY_DEVICE_ACTIVE != 0 {
                connected_monitors += 1;
            }
            i += 1;
        }
    }
    connected_monitors as i32
}

[H-M-H]

Sadly captrs implements a function to get the offset for linux but not for windows

Oh well, I looked into the docs and saw a position method but the docs are generated on a linux machine and it indeed looks like it's missing for windows...

But the good news is that it looks like directly calling winapi can solve this:
https://github.com/bryal/dxgcap-rs/blob/009b746d1c19c4c10921dd469eaee483db6aa002/src/lib.rs#L270-L275
As far as I can tell this is calling this method: https://docs.microsoft.com/en-us/windows/win32/api/dxgi/nf-dxgi-idxgioutput-getdesc which I mentioned earlier. It requires getting the displays through EnumOutputs though.

Calculating the offsets is weird though since: rcMonitor.left * -1 + event.x results in a correct coordinate for "Monitor 1" but i dont know how that maps to more monitors with different layouts than left->right.

I wouldn't go there, that indeed sounds like different layouts would immediately break this.

Btw this doesnt return the same order as the EnumMonitorDevices method. Main Display is 1 with EnumMonitorDevices and 0 with EnumDisplayDevicesA for whatever reason

I'd say getting EnumOutputs working instead of relying on the other two functions is the best option, even though it sadly looks like it's going to be a little more complicated.

I have a working setup for 1 monitor though :)

Awesome, that's great progress!

[qdlmcfresh]

You are right that using EnumDisplays will probably be the easiest way to get the RECT of each Monitor indexed in a way that captrs will understand. Thats not really my problem... Sorry if i failed to communicate that in my previous reply, i was in a rush when writing that.
So ill try again 😄 :

This is one possible virtual screen layout. For winapi functions like mouse_event or SetCursorPos(x, y), the coordinate system in the picture is used. so (0,0) will end up to be the top left corner of the primary display and (0 - Monitor2.width, 0) will be the top left corner of Monitor2. The RECT you get by GetMonitorInfo/iDXGIOUTPUT.getDesc will work perfectly fine for these functions since the RECT for Monitor2 will be

{
left: -1 * (MON2_WIDTH)
top: 0
right: 0
bottom: MON2_HEIGHT
}

For InjectSyntheticPointerInput this is not the case though. MSDOC: "The ptPixelLocation for each POINTER_TYPE_INFO is specified relative to top left of the virtual screen" So (0,0) will end up to be the top-left corner of Monitor2 in the picture above.

What im struggeling to wrap my head around is an easy way to calculate the actual coordinate of the top-left corner of each monitor on the virtualdisplay-coordinatesystem as its understood by InjectSyntheticPointerInput

[H-M-H]

Oh, I see.
Are the values for bottom and top really correct in the example you gave for monitor 2, I'd expect some offset so it matches the picture.

If there is indeed some kind of offset the following should be correct: The coordinate systems only differ by a translation, so to go from getDesc to Virtual Screen you just need to subtract the minimal left value over all monitors as well as the minimal top value respectively for left and right, top and bottom of the monitor coordinates you want to convert.

[qdlmcfresh]

If there is indeed some kind of offset the following should be correct: The coordinate systems only differ by a translation, so to go from getDesc to Virtual Screen you just need to subtract the minimal left value over all monitors as well as the minimal top value respectively for left and right, top and bottom of the monitor coordinates you want to convert.

Yes i think thats right, i was testing just now and came up with this:

    void print() {
        int i = 0;
        for (const RECT &r : this->rcMonitors) {
            std::printf("Monitor[%i] Left: %i Top: %i  Right: %i Bottom: %i\n", i++, r.left, r.top, r.right,r.bottom);
        }
        std::printf("Union: Left: %i Top: %i  Right: %i Bottom: %i\n", this->rcCombined.left , this->rcCombined.top, this->rcCombined.right, this->rcCombined.bottom );
        std::printf("\n\n");
        for (const RECT& r : this->rcMonitors) {
            std::printf("Monitor+Offset[%i] Left: %i Top: %i  Right: %i Bottom: %i \n", i++,r.left - rcCombined.left, r.top,  r.right + rcCombined.right, r.bottom);
        }   
    }

Monitor[0] Left: -1920 Top: 0  Right: 0 Bottom: 1080
Monitor[1] Left: 0 Top: 0  Right: 2560 Bottom: 1440
Monitor[2] Left: 0 Top: -1080  Right: 1920 Bottom: 0

Union: Left: -1920 Top: -1080  Right: 2560 Bottom: 1440

Monitor+Offset[0] Left: 0 Top: 0  Right: 2560 Bottom: 1080
Monitor+Offset[1] Left: 1920 Top: 0  Right: 5120 Bottom: 1440
Monitor+Offset[2] Left: 1920 Top: -1080  Right: 4480 Bottom: 0

(right and bottom wont matter i think so i ignored them)

For this Screen layout:

[H-M-H]

That's looking good, it appears top and bottom behave as I expected.

I do think instead of

std::printf("Monitor+Offset[%i] Left: %i Top: %i  Right: %i Bottom: %i \n", i++,r.left - rcCombined.left, r.top,  r.right + rcCombined.right, r.bottom);

it should be

std::printf("Monitor+Offset[%i] Left: %i Top: %i  Right: %i Bottom: %i \n", i++,r.left - rcCombined.left, r.top - rcCombined.top,  r.right - rcCombined.left, r.bottom - rcCombined.top);

though.

[qdlmcfresh]

Ok cool i think we have this worked out.

use std::mem::zeroed;
use std::{mem, ptr};
use winapi::shared::dxgi::{
    CreateDXGIFactory1, IDXGIAdapter1, IDXGIFactory1, IDXGIOutput, IID_IDXGIFactory1,
    DXGI_OUTPUT_DESC,
};

use winapi::shared::windef::*;
use winapi::shared::winerror::*;
use winapi::um::winuser::*;
use wio::com::ComPtr;

// from https://github.com/bryal/dxgcap-rs/blob/009b746d1c19c4c10921dd469eaee483db6aa002/src/lib.r
fn hr_failed(hr: HRESULT) -> bool {
    hr < 0
}

fn create_dxgi_factory_1() -> ComPtr<IDXGIFactory1> {
    unsafe {
        let mut factory = ptr::null_mut();
        let hr = CreateDXGIFactory1(&IID_IDXGIFactory1, &mut factory);
        if hr_failed(hr) {
            panic!("Failed to create DXGIFactory1, {:x}", hr)
        } else {
            ComPtr::from_raw(factory as *mut IDXGIFactory1)
        }
    }
}

fn get_adapter_outputs(adapter: &IDXGIAdapter1) -> Vec<ComPtr<IDXGIOutput>> {
    let mut outputs = Vec::new();
    for i in 0.. {
        unsafe {
            let mut output = ptr::null_mut();
            if hr_failed(adapter.EnumOutputs(i, &mut output)) {
                break;
            } else {
                let mut out_desc = zeroed();
                (*output).GetDesc(&mut out_desc);
                if out_desc.AttachedToDesktop != 0 {
                    outputs.push(ComPtr::from_raw(output))
                } else {
                    break;
                }
            }
        }
    }
    outputs
}

pub struct win_ctx {
    outputs: Vec<RECT>,
    union_rect: RECT,
}

impl win_ctx {
    pub fn new() -> win_ctx {
        let mut rects: Vec<RECT> = Vec::new();
        let mut union: RECT;
        unsafe {
            union = mem::zeroed();
            let factory = create_dxgi_factory_1();
            let mut adapter = ptr::null_mut();
            if factory.EnumAdapters1(0, &mut adapter) != DXGI_ERROR_NOT_FOUND {
                let adp = ComPtr::from_raw(adapter);
                let outputs = get_adapter_outputs(&adp);
                for o in outputs {
                    let mut desc: DXGI_OUTPUT_DESC = mem::zeroed();
                    o.GetDesc(ptr::addr_of_mut!(desc));
                    rects.push(desc.DesktopCoordinates);
                    UnionRect(
                        ptr::addr_of_mut!(union),
                        ptr::addr_of!(union),
                        ptr::addr_of!(desc.DesktopCoordinates),
                    );
                }
            }
        }
        win_ctx {
            outputs: rects,
            union_rect: union,
        }
    }
    pub fn get_outputs(&self) -> &Vec<RECT> {
        &self.outputs
    }
    pub fn get_union_rect(&self) -> &RECT {
        &self.union_rect
    }
}

What would be the best way to get the current screen_width, screen_height, offset_x, offset_y back to the InputDevice?

[H-M-H]

Looks like the input device needs to know about the window context too, so I guess it can't be helped but to lookup the dimensions of the Virtual Screen inside the input device with its own handle as well. Then the Capturables geometry_relative can be translated back to what InjectSyntheticPointerInput expects.

Perhaps a little explanation why I designed things like this: On Linux you can simulate inputs on a pretty low level with absolutely no knowledge of display sizes or positions. You can even declare your own (rectangular) input coordinate system which on some higher level is then mapped to actual screen coordinates. InjectSyntheticPointerInput lives on a higher level and is aware of the Virtual Screen and that's why things are getting a little awkward here.

[qdlmcfresh]

It works now with multiple monitors, i edited the capturable trait though. I dont think thats the perfect solution, but i wanted to see if the coordinate translation is correct before figuring the rest out

Just tell me what i have to change for it to get merged and i will try my best :)

[H-M-H]

I'd say lets get it building everywhere, fixup the trait and then I will have a look again. Thanks for your work!

[qdlmcfresh]

I'd say lets get it building everywhere, fixup the trait and then I will have a look again. Thanks for your work!

Ragarding fixing up the trait. While researching the Box pattern i found this post and if i understand it correctly this fits here. Since all of the Capturable implementations hold different data, that might be needed in the implementations of Input but there is no way to get back to the original struct after putting it in a box.

[H-M-H]

How about a compromise, I still think Box<dyn _> makes sense here and I'd like to avoid leaking too much unnecessary details about the capturable. So what I think should work is to change the Capturable trait, so it has one geometry method that returns an enum of possible Geometries:

enum Geometry {
    Relative(f64, f64, f64, f64),
    VirtualScreen(i32, i32, u32, u32),
}

The VirtualScreen variant can then be marked unreachable!() on non windows systems.

[qdlmcfresh]

Sounds good

[H-M-H]

Any progress regarding touch input? Can I help somehow?

[qdlmcfresh]

Sry dont have too much spare time at the moment. The only thing missing regarding touch input is correct handling of multitouch. I do get multiple pointers for multiple fingers but they dont work together. Another thing i had a deeper look at was this. I was able to confirm that window-capturing is possible like its done there, but i wasnt able to integrate it with weylus yet.

Multitouch looks like this atm:

[H-M-H]

Thanks for the update!

Sry dont have too much spare time at the moment

Nothing to worry about, I mean we all do this in our spare time. Also have a merry Christmas 🎄!

[qdlmcfresh]

Nothing to worry about, I mean we all do this in our spare time. Also have a merry Christmas 🎄!

Thank you 😃, have nice holidays aswell 🎅

[qdlmcfresh]

First of all happy new year! :) Regarding display and window capture again. I asked the author of displayrecorderand screenshot-rs if its is possible to get a unencoded frame the way hes capturing in displayrecorder here. The answer was sadly, no. But since hes using mediafoundation aswell maybe this could still be usefull. I just donk know enough about directx and mediafoundation to judge that.

[H-M-H]

@qdlmcfresh how's this going, will you have time in the foreseeable future? If not, perhaps somebody else using Windows as well is willing to take this over and get multi touch working.

[qdlmcfresh]

@qdlmcfresh how's this going, will you have time in the foreseeable future? If not, perhaps somebody else using Windows as well is willing to take this over and get multi touch working.

Time isnt even the issue here. I just dont know how to solve the problem with multitouch ... So if anybody has something to add to this they are very welcome to do so. :)

[ghost]

@qdlmcfresh how's this going, will you have time in the foreseeable future? If not, perhaps somebody else using Windows as well is willing to take this over and get multi touch working.

Time isnt even the issue here. I just dont know how to solve the problem with multitouch ... So if anybody has something to add to this they are very welcome to do so. :)

#156 fixed the issue.

[qdlmcfresh]

Since @Ph-Fr-One is no longer with us i took the liberty to integrate his changes into this PR. I tested with 2 screens on Win10 and zoom/pinch gestures are working fine.

[aegistudio]

I've seen that captrs is constantly reporting CaptureError::Timeout when I am using. I've tried tweaking it by specifying a timeout (e.g. Capturer::new_with_timeout(id.into(), Duration::from_millis(1000))) but it did not work for me.

Are these just some expected and okay-to-ignore errors when working with captrs, or do I need some further configurations?

Update: it turns out to live happily with these errors, so I think I could safely ignore them.

[Etran-H]

Excuse me, any problem with this PR? @H-M-H

[flomorky]

Hello hello guys i'v just read your topic and it soo cool, but im little pickle with all does stuff and I don't understand any of it. i'm trying to understand it but its hard. So did you finally manage to add pointer movment with weylus ? I would really love to use it if you could show me how to install it. Cause I got now but it does not work with pen sensivity of my Ipad :/

[qdlmcfresh]

Hello hello guys i'v just read your topic and it soo cool, but im little pickle with all does stuff and I don't understand any of it. i'm trying to understand it but its hard. So did you finally manage to add pointer movment with weylus ? I would really love to use it if you could show me how to install it. Cause I got now but it does not work with pen sensivity of my Ipad :/

https://github.com/kotaewing/Weylus/releases/tag/V1.1 should be a build with the changes in this PR integrated.

[flomorky]

Wow thats super man thanks ! Now the only thing would be to add a wired connection to it so there will be no latency. Its like almost the perfect software but trying to sketch on Photoshop with it is a bit tricky with the delay Le jeu. 2 mars 2023, 03 h 44, qdlmcfresh ***@***.***> a écrit :

…

Hello hello guys i'v just read your topic and it soo cool, but im little pickle with all does stuff and I don't understand any of it. i'm trying to understand it but its hard. So did you finally manage to add pointer movment with weylus ? I would really love to use it if you could show me how to install it. Cause I got now but it does not work with pen sensivity of my Ipad :/ https://github.com/kotaewing/Weylus/releases/tag/V1.1 should be a build with the changes in this PR integrated. — Reply to this email directly, view it on GitHub <#128 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A6G24TG2SIQDM3G7IJCNPVLW2BMXZANCNFSM5IZEFYBQ> . You are receiving this because you commented.Message ID: ***@***.***>

[digitalsignalperson]

I got this to build with the hhmhh/weylus_build docker image by

• extracting ffmpeg-n6.0-latest-win64-gpl-shared-6.0.zip from https://github.com/BtbN/FFmpeg-Builds into /Weylus/deps/dist_windows
• it doesn't include libx264 though, so grabbed libx264_0.164.r3107_msvc17.zip from https://github.com/ShiftMediaProject/x264/releases

docker run -it hhmhh/weylus_build bash
git clone https://github.com/qdlmcfresh/Weylus.git -b stylus_windows

# from host
docker cp ~/x/ffmpeg-n6.0-latest-win64-gpl-shared-6.0/. 878bc718f089:/Weylus/deps/dist_windows/.
docker cp ~/x/libx264_0.164.r3107_msvc17/bin/x64/x264.dll 878bc718f089:/Weylus/deps/dist_windows/bin/
docker cp ~/x/libx264_0.164.r3107_msvc17/include/x264.h 878bc718f089:/Weylus/deps/dist_windows/include/
docker cp ~/x/libx264_0.164.r3107_msvc17/include/x264_config.h 878bc718f089:/Weylus/deps/dist_windows/include/
docker cp ~/x/libx264_0.164.r3107_msvc17/lib/x64/libx264.lib 878bc718f089:/Weylus/deps/dist_windows/lib/
docker cp ~/x/libx264_0.164.r3107_msvc17/lib/x64/x264.lib 878bc718f089:/Weylus/deps/dist_windows/lib/

# in container
cargo build --target x86_64-pc-windows-gnu --release

# from host
docker cp 878bc718f089:/Weylus/target/x86_64-pc-windows-gnu/release/weylus.exe ~/x/

Also for weylus.exe to work I had to bundle it with all the dll files from the bin folders from those zips.

Very cool this works.

This message is constantly spammed in the log, but I didn't notice an impact: WARN Error capturing screen: Captrs failed to capture frame

[digitalsignalperson]

Does this work for anyone with server = windows, client = linux? If not I bet it'd be pretty close based on this work so far.

For me I can do windows to windows, but when I try remoting to a windows machine from linux, the stylus events don't show up on the windows side. And I can confirm with libinput debug-events that I do have stylus events on the linux source.

This would be super awesome to have a weylus server in a windows VM and be able to use the stylus from a a linux host. I don't think there's any other solution right now to do linux -> windows stylus stuff. If anyone knows of anything I'd love to hear.

For windows to windows, RDP already supports pen & pressure and touch. But this isn't implemented in freerdp yet FreeRDP/FreeRDP#7260

[digitalsignalperson]

I also just tested that windows <--> windows RDP shadowing work with the pen and pressure too. Shadowing is screen is mirrored on both sides without RDP taking over, so very similar to this PR but built into windows. Also unfortunately this is not implemented in freerdp for linux --> windows FreeRDP/FreeRDP#5656

To try the rdp shadowing, see group policy for Computer Configuration -> Administrative Templates -> Windows components -> Remote Desktop Services -> Remote Session Host -> Connections -> Set rules for remote control of Remote Desktop Services user sessions

and then

mstsc /shadow:1 /v:192.168.0.xx /prompt /noConsentPrompt /control

Oh, with the limiting factor of needing Pro editions of windows which sucks.

I'll try to look at the linux --> windows possibilities with this PR later