Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2.9.0 suddenly causes poor connection to cameras #1193

Open
vipergts450 opened this issue May 13, 2024 · 16 comments
Open

2.9.0 suddenly causes poor connection to cameras #1193

vipergts450 opened this issue May 13, 2024 · 16 comments

Comments

@vipergts450
Copy link

vipergts450 commented May 13, 2024

Hello,

I upgraded to 2.9.0 yesterday and suddenly the connections from the bridge to my two cameras is very unstable. The log has these messages every couple of seconds. I have ON_DEMAND=False set in my config but that did not help. Nothing about my configuration changed since the prior version which had been rock solid.

Possible related to this person's issue: #1186 (comment)

2024-05-13T16:13:03.196702946Z  * Serving Flask app 'frontend'
2024-05-13T16:13:03.196970160Z [WyzeBridge] 📚 Using 'auth' from local cache...
2024-05-13T16:13:03.197045726Z  * Debug mode: off
2024-05-13T16:13:03.198131787Z [WyzeBridge] 📚 Using 'user' from local cache...
2024-05-13T16:13:03.199037258Z [WyzeBridge] 📚 Using 'cameras' from local cache...
2024-05-13T16:13:03.199659505Z [WyzeBridge] 🪄 FILTER ALLOWING: 2 of 3 cams
2024-05-13T16:13:03.200873864Z [WyzeBridge] [+] Adding cam1 [WYZE_CAKP2JFUS]
2024-05-13T16:13:03.203321704Z [WyzeBridge] WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
2024-05-13T16:13:03.203474466Z  * Running on all addresses (0.0.0.0)
2024-05-13T16:13:03.203525782Z  * Running on http://127.0.0.1:5000
2024-05-13T16:13:03.203571620Z  * Running on http://172.23.0.2:5000
2024-05-13T16:13:03.203615671Z [WyzeBridge] Press CTRL+C to quit
2024-05-13T16:13:03.215488053Z [WyzeBridge] [+] Adding cam2 [WYZECP1_JEF]
2024-05-13T16:13:03.216297232Z [WyzeBridge] starting MediaMTX 1.1.1
2024-05-13T16:13:03.217069413Z [WyzeBridge] 🎬 2 streams enabled
2024-05-13T16:13:04.205027178Z [WyzeBridge] 172.23.0.1 - - [13/May/2024 16:13:04] "GET /api/sse_status HTTP/1.1" 200 -
2024-05-13T16:13:04.218120477Z [WyzeBridge] 🎉 Connecting to WyzeCam V3 - cam1 on 192.168.2.52
2024-05-13T16:13:04.245859903Z [WyzeBridge] 🎉 Connecting to WyzeCam Pan - cam2 on 192.168.2.50
2024-05-13T16:13:06.272797673Z [cam1] 📡 Getting 180kb/s HD stream (H264/20fps) via LAN mode (WiFi: 92%) FW: 4.36.11.8391 🔒
2024-05-13T16:13:06.273624367Z [cam1] 🔊 Audio Enabled - ALAW > AAC/16,000Hz
2024-05-13T16:13:06.351398248Z [cam2] 📡 Getting 180kb/s HD stream (H264/15fps) via LAN mode (WiFi: 97%) FW: 4.10.9.3006 🔒
2024-05-13T16:13:06.352678972Z [cam2] 🔊 Audio Enabled - MULAW > AAC/8,000Hz
2024-05-13T16:13:06.370007139Z [cam1] WARNING: Skipping wrong frame_size at start of stream [frame_size=1]
2024-05-13T16:13:06.446627324Z [cam2] WARNING: Skipping wrong frame_size at start of stream [frame_size=1]
2024-05-13T16:13:06.980112380Z [WyzeBridge] ✅ '/cam2 stream is UP! (3/3)
2024-05-13T16:13:07.438466197Z [WyzeBridge] 📖 New client reading from cam2
2024-05-13T16:13:08.033675802Z [WyzeBridge] ✅ '/cam1 stream is UP! (3/3)
2024-05-13T16:13:08.153190291Z [WyzeBridge] 📖 New client reading from cam2
2024-05-13T16:13:08.448102057Z [WyzeBridge] 📖 New client reading from cam1
2024-05-13T16:13:08.615706720Z [WyzeBridge] 📕 Client stopped reading from cam2
2024-05-13T16:13:09.649123751Z [WyzeBridge] 📖 New client reading from cam1
2024-05-13T16:13:10.017194505Z [WyzeBridge] 📕 Client stopped reading from cam1
2024-05-13T16:13:10.422995827Z [WyzeBridge] 📖 New client reading from cam1
2024-05-13T16:13:11.232538143Z [WyzeBridge] 📕 Client stopped reading from cam1
Config lines:
version: '2.4'
services:
    wyze-bridge:
        container_name: wyze-bridge
        restart: unless-stopped
        image: mrlt8/wyze-bridge:latest
        ports:
            - 1936:1935 # RTMP
            - 8555:8554 # RTSP
            - 8888:8888 # HLS
            - 8889:8889 #WebRTC
            - 8189:8189/udp # WebRTC/ICE
            - 8002:5000 # WEB-UI
        environment:
            # [OPTIONAL] (Can be set in the WebUI):
            - WYZE_EMAIL=xxxx
            - WYZE_PASSWORD=xxxx
            # [OPTIONAL] IP Address of the host to enable WebRTC e.g.,:
            - WB_IP=192.168.2.4
            - WB_API=xxxx
            # [OPTIONAL] Wyze API credentials:
            - API_ID=xxxx
            - API_KEY=xxxx
            - ENABLE_AUDIO=True
            - AUDIO_CODEC=AAC
            - FILTER_NAMES=cam1, cam2
            - SUBSTREAM=False
            - NET_MODE=LAN
            - ON_DEMAND=False

Could this be a port mapping issue that arose with the latest update? I don't see any log lines that suggest the bridge is losing the feed from the cameras, but seemingly clients connecting externally to the bridge to retrieve the feeds have this constant connect/disconnect problem.

The choppy stream, timeout when fetching snapshot on the WebUI (the auto refresh sometimes pulls a new snapshot image and sometimes does not for 15-20min), and general misbehavior seems to point to something between the front-end and the mediamtx backend, but I can't seem to figure out what exactly.

@1fastt2
Copy link

1fastt2 commented May 13, 2024

I am experiencing the same thing. I reverted back to 2.8.3 which has been solid. I did confirm I have the same issues with 2.9.1 as well.

@vipergts450
Copy link
Author

vipergts450 commented May 13, 2024

This doesn't have anything to do with the new FFMPEG option -use_wallclock_as_timestamps I presume? Could this option cause issues with NVRs? Possibly better config option could be -live_flv according to the Frigate forums. However, this is possibly also the entirely wrong rabbithole to go down.

@mrlt8
Copy link
Owner

mrlt8 commented May 14, 2024

Could you try the latest dev build to see if it helps?

@vipergts450 it's a timestamp issue with the raw h264 - the cameras return a timestamp together with the raw h264, but there doesn't seem to be a way to pass that info on to ffmpeg so use_wallclock_as_timestamps seems to help keep everything in sync.

mrlt8 added a commit that referenced this issue May 14, 2024
@vipergts450
Copy link
Author

On the dev branch as of this morning. Will report back in a bit.

@vipergts450
Copy link
Author

vipergts450 commented May 14, 2024

It's certainly more stable than yesterday, which is a big help. However, something's still going on with the timestamping or keyframing on my Cam v3 (oddly not happening with my very old OG Cam Pan). Do these cams handle the stream keying differently?

2024-05-14T12:00:09.784351760Z [cam1] [video] super slow
2024-05-14T12:00:09.785032712Z [cam1] WARNING: clear buffer
2024-05-14T12:01:54.293497646Z [cam1] WARNING: Audio pipe closed
2024-05-14T12:01:54.994294824Z [WyzeBridge] 📕 Client stopped reading from cam1
2024-05-14T12:01:55.452390762Z [cam1] [Exception] Did not receive a frame for 39s
2024-05-14T12:01:59.528215199Z [WyzeBridge] ❌ '/cam1' stream is down
2024-05-14T12:01:59.528750219Z [WyzeBridge] 🎉 Connecting to WyzeCam V3 - Cam1 on 192.168.2.52
2024-05-14T12:02:01.743734438Z [cam1] 📡 Getting 180kb/s HD stream (H264/15fps) via LAN mode (WiFi: 93%) FW: 4.36.11.8391 🔒
2024-05-14T12:02:01.744719617Z [cam1 🔊 Audio Enabled - ALAW > AAC/16,000Hz
2024-05-14T12:02:01.844483893Z [cam1] WARNING: Skipping wrong frame_size at start of stream [frame_size=1]
2024-05-14T12:02:02.107081553Z [WyzeBridge] ✅ '/cam1 stream is UP! (3/3)
2024-05-14T12:02:10.485513162Z [WyzeBridge] 📖 New client reading from cam1

Once the stream came back up on its own, it's stable again for the moment. It looks like the stream was up for almost exactly 30 minutes before resetting.

Update: happened again after exactly 30 minutes just as before. So looks like there's a 30 min cycle for resets on the Cam v3 at least, maybe as a counter rolls over? My Cam Pan stream has not reset at all since the container went online.

@mrlt8
Copy link
Owner

mrlt8 commented May 14, 2024

hmmm, thank you for the data point! I wonder if the camera's might be running some kind of clock sync every 30 minutes that might be throwing off our av sync checks....

@vipergts450
Copy link
Author

vipergts450 commented May 14, 2024

Updating again. Since this morning, the Cam V3 has done the 30 min reset consistently, but it's now fast enough that the NVR it's connected to doesn't "notice" the camera going down every single time (only maybe 3 times total all day did it notice the disconnect and send me a warning email) and quickly resumes when it comes back up. The old Cam Pan has never dropped. So the two firmwares seem to do keyframing/timestamping/time sync a bit differently. This current dev version seems to be a reasonable stopgap fix in the interim as compared to 2.9.0-2.9.1 in my opinion until a more permanent fix is discovered.

@vipergts450
Copy link
Author

vipergts450 commented May 14, 2024

Update: Stream for the Cam v3 finally went kaput exactly 9 hours, 21min, 25 seconds after starting this morning and did not recover. That's about 33,685 seconds or ~15 min after a signed 16-bit integer counter would roll over past 32768. Not sure if coincidental or related some way how the new FFMPEG counter works. It did not recover again as it had throughout the day and remains permanently down.

2024-05-14T20:31:21.276990825Z [WyzeBridge] ✅ '/cam1 stream is UP! (3/3)
2024-05-14T20:31:29.356919469Z [WyzeBridge] 📖 New client reading from cam1
2024-05-14T20:50:45.169324471Z [cam1] [video] super slow
2024-05-14T20:50:45.170020970Z [cam1] WARNING: clear buffer
2024-05-14T20:51:24.367778977Z [cam1] [video] super slow
2024-05-14T20:51:51.152894206Z [cam1] [video] super slow
2024-05-14T20:52:14.384863744Z [WyzeBridge] 📕 Client stopped reading from cam1
2024-05-14T20:52:15.210762472Z [cam1] WARNING: Audio pipe closed
2024-05-14T20:52:15.774693428Z [cam1] Stream stopped

That's the log from the last "normal" reset I saw all day until the final stream ended.

The Cam Pan (cam2) has stayed up all day and never had a "blip".

@vipergts450
Copy link
Author

Today's dev release has kept both streams up since starting up 4 hours ago.

@1fastt2
Copy link

1fastt2 commented May 15, 2024

I can also confirm that the latest DEV build is solid in regards to the original issue of the thread. I still have zero success with V4 cameras but I don't want to get off topic of this thread.

@mrlt8
Copy link
Owner

mrlt8 commented May 16, 2024

Unfortunately, it seems like the audio seems to drift behind on the dev branch.

@vipergts450
Copy link
Author

vipergts450 commented May 16, 2024

How about trying the live_flv flag?

This demuxer is used to demux FLV files and RTMP network streams. In case of live network streams, if you force format, you may use live_flv option instead of flv to survive timestamp discontinuities.

Also, this issue with the 13hr 15min bug I mentioned a few days ago. Maybe I did my math wrong when I said 9 hours?

That thread references using -correct_ts_overflow 0 as an input flag potentially.

bluenviron/mediamtx#622

And later in
bluenviron/mediamtx#922

Looks like it was fixed in MediaMTX in 0.18.4?

mrlt8 added a commit that referenced this issue May 16, 2024
@mrlt8
Copy link
Owner

mrlt8 commented May 16, 2024

Thanks @vipergts450, I will have to take a look at those.

I've added the -re flag for the audio which seems to help keep a stable connection and the audio in sync for a couple of hours now.

Would appreciate any feedback!

@vipergts450
Copy link
Author

I am pulling the dev now as of this writing and will report back.

@mrlt8
Copy link
Owner

mrlt8 commented May 17, 2024

@vipergts450 Any feedback on the last dev build?

@vipergts450
Copy link
Author

@mrlt8 I have a full day under my belt on this latest dev build and have received no disconnects from the NVRs connected to the stream, and the logs only give the occasional [cam1] [video] super slow. Cam 2 has never been an issue through all these dev builds (older firmware, older camera). I have not paid attention to the audio delays since I don't make heavy use of that, so unfortunately I am a bad data point for this test for audio sync.

mrlt8 added a commit that referenced this issue May 17, 2024
* Tweak AV sync and ffmpeg cmd #1175 #1196 #1194 #1193 #1186

* Catch AccessTokenError

* don't drop timestamp #1175 #1196 #1194 #1193 #1186

* Don't use_wallclock_as_timestamps  #1175 #1196 #1194 #1193 #1186

* keep audio in sync #1175 #1196 #1194 #1193 #1186

* Ignore whitespaces in api key/id #1188

* Remove quotes from credentials #1158

* HA Add FORCE_FPS option #1161

* FORCE_FPS option for all cameras #1161

* changelog
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants