[ota] http_request update platform #5586

oarcher · 2023-10-22T23:33:22Z

What does this implement/fix?

ota in client (http) mode: This allow to do a remote ota when the device is far away, behind a firewall, and only reachable with mqtt. (For ex in another house).

As the ota might be triggered by mqtt, a long time deep sleep can be defined.

original component from https://github.com/oarcher/piotech/tree/main/components/ota_http

Types of changes

Bugfix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Other

Related issue or feature (if applicable): fixes esphome/feature-requests#801

Pull request in esphome-docs with documentation (if applicable): esphome/esphome-docs#3291

Test Environment

ESP32
ESP32 IDF (v4 & v5)
ESP8266
RP2040

Http/Https support matrix

	ESP32	ESP32 IDF	ESP8266	RP2040
HTTP	Yes	Yes	Yes	Yes
HTTPS	Yes	Yes	Yes	Yes

Example entry for `config.yaml`:

Example config.yaml

ota_http:


button:
  - platform: template
    name: "Firmware update"
    on_press:
      then:
        - ota_http.flash:
            url: http://example.com/firmware.bin
        - logger.log: "This message should be not displayed(reboot)"

Checklist:

The code change is tested and works locally.
Tests have been added to verify that the new code works (under tests/ folder).

If user exposed functionality or configuration variables are added/changed:

Documentation added/updated in esphome-docs.

probot-esphome · 2023-10-22T23:33:32Z

Hey there @oarcher,
Thanks for submitting this pull request! Can you add yourself as a codeowner for this integration? This way we can notify you if a bug report for this integration is reported.
In __init__.py of the integration, please add:

CODEOWNERS = ["@oarcher"]

And run script/build_codeowners.py

_{^{(message by NeedsCodeownersLabel)}}

oarcher · 2023-10-23T21:26:35Z

I can see that PR #5535 change the implementation of OTA backend in esphome/components/ota/ota_backend*.
So some parts of this PR should be rewritten once #5535 as been merged.

error_code = esphome::ota_http::OtaHttpComponent::BACKEND->begin(this->body_length_);

should become:

error_code = esphome::ota_http::OtaHttpComponent::BACKEND->begin(OTA_BIN_APP, this->body_length_);

nerijus · 2023-10-25T12:13:42Z

"https only supported for esp32" - does it mean that https is supported on esp32 only, not other platforms, or that esp32 supports https only, no http for example?

oarcher · 2023-10-26T23:56:42Z

Sorry for the confusion. https and http works for esp32. I've edited the PR description with http/https support matrix.

oarcher · 2023-10-27T02:12:52Z

@nerijus Previous commit has fixed https for 8266.

nerijus · 2023-10-27T08:35:16Z

@nerijus Previous commit has fixed https for 8266.

So the line "https only supported for esp32" can probably be removed at all?

angelnu · 2023-11-19T19:22:11Z

@oarcher - following with our conversation - can you please take a look at the split of the OTAComponent I drafted at angelnu#1

Basically splitted the socket usage as frontend (see ota_socket component). It compiles but I did not test it yet - if this would also work for http I would clean up the code and take care of other issues

UPDATE
I tested it and it works - I can update via socket OTA after the split.

Alphaemef · 2023-12-23T01:17:16Z

Man I have to say again @oarcher. This thing is a beast. Probably used it a couple of hundred times by now. Its amazing. Good job man.

codecov-commenter · 2024-01-11T20:20:14Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 53.90%. Comparing base (4d8b5ed) to head (5a98daa).
Report is 719 commits behind head on dev.

Additional details and impacted files

@@            Coverage Diff             @@
##              dev    #5586      +/-   ##
==========================================
+ Coverage   53.70%   53.90%   +0.19%     
==========================================
  Files          50       50              
  Lines        9408     9623     +215     
  Branches     1654     1698      +44     
==========================================
+ Hits         5053     5187     +134     
- Misses       4056     4112      +56     
- Partials      299      324      +25

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

kevireilly · 2024-02-13T09:43:24Z

Possible to support the advanced updater which allows the server to determine if there should be an OTA update performed? Useful for automatic updates or pushing specific updates to specific clients depending on the version string passed.

Wasn't able to find official up to date documentation, but am remembering something along these lines from years ago: https://arduino-esp8266.readthedocs.io/en/latest/ota_updates/readme.html#advanced-updater

paulsteigel · 2024-02-21T17:45:24Z

Hi All,
Many thanks @oarcher and team for your works on ota_http that is brilliant. I found this with similar feature but it can do the OTA over MQTT.
If possible, could you please be so kind to help the community to do the OTA over MQTT using this approach.
Many thanks for the ESPHOME team.
Yours very faithfully,
Dang Dinh Ngoc

oarcher · 2024-02-21T20:29:11Z

@paulsteigel , ota_http works well with mqtt, you just have to have a mqtt broker defined in an mqtt: section in your yaml. ota.http.flash is just an action, that can be triggered by any events, including mqtt.

mqtt:
  broker: mqttbrocker.example.com
  username: mqtt_username
  password: mqtt_password

button:
  - platform: template
    name: "Firmware update"
    on_press:
      then:
        - ota_http.flash:
            url: http://example.com/firmware.bin
            verify_ssl: false
        - logger.log: "This message should be not displayed(reboot)"

oarcher · 2024-02-21T20:34:56Z

@kevireilly , yes, I plan to add this feature to ota_http. This will be done with the Arduino/NoFuss protocol (see esphome/feature-requests#801 (comment)).

But for now, I'm waiting for the main esphome maintainers to merge this PR.

paulsteigel · 2024-02-22T04:22:37Z

Hi @oarcher,
Many thanks for your quick responses! I use your ota_http but it seems that there a bit of problem (can be due to my limited knowledge) and the update failed as following:
I tried your component and it throw the following error:

[21:38:09][D][button:010]: 'Firmware update' Pressed.
[21:38:09][D][esp32.preferences:114]: Saving 1 preferences to flash...
[21:38:09][D][esp32.preferences:143]: Saving 1 preferences to flash: 0 cached, 1 written, 0 failed
[21:38:09][D][ota_http:054]: Trying to connect to https://mywebpage.com.vn/firmwares/[mydeviceddns]/firmware.bin
[21:38:12][D][ota_http:101]: firmware is 1132272 bytes length.
[21:38:12][I][ota_http:111]: Progress: 0.1%
[21:38:12][E][ota_http:103]: Error code (131) writing binary data to flash at offset 4000 and size 1132272
[21:38:12][E][ota_http:150]: Abort OTA backend
[21:38:12][E][ota_http:153]: Abort http con
[21:38:12][E][ota_http:155]: previous safe mode didn't succed. ota_http skipped
[21:38:12][D][main:603]: This message should be not displayed(reboot)

My esphome configuration is here

ota_http:

button:
  - platform: template
    name: "Firmware update"
    on_press:
      then:
        - ota_http.flash:
            url: https://mywebpage.com.vn/firmwares/mydeviceddns/firmware.bin # revise it here
            verify_ssl: false
        - logger.log: "This message should be not displayed(reboot)"

My webpage.com.vn is the the replacement of my real address to avoid security risks. It shown that firmware was download successfully but failed in installing.
Regarding OTA over MQTT, from my knowledge, firmware build by Esphome in esphome docker (that i am now have to do the download manually, and this is a bit inconvenient rather than just click the install then all will do automatically) should be:

Encoded to base64 or whatever way then publish at an MQTT topic;
Trigger an action for the esphome device to download and install as it currently does via ota/8266.
It would be nice that we just need to click Install then firmware will be built and pass on MQTT protocol for device download and install as it is now over port 8266.
Yours very faithfully,
Dang DInh Ngoc

Co-authored-by: Jesse Hills <3060199+jesserockz@users.noreply.github.com>

paulsteigel · 2024-06-05T16:20:54Z

Hi @kbx81 and @oarcher,
I did a new HA installation with ESPHOME dev and continue the test above. My HA sit behind a draytek Vigor2920 router with port 443 forwarded to the HA server (that is listening on 443 with Let'sencrypt certificate). Testing of files (md5 and bin) is working fine with browser.
However, when I click the button, the following error appear.

23:08:29][I][http_request.ota:268]: Connecting to: https://myhome.sfdp.net/local/tunglan1976/firmware.md5
[23:08:29][E][http_request.ota:074]: HTTP server did not respond (error -1)

I used the config from @kbx81 and the compiler complained that "force_update is invalid option for ota_http_request.flash:"

ota:
  - platform: http_request
    exclude_certificate_bundle: true
    on_state_change:
      - logger.log: http_request.on_state_change
    on_abort:
      - logger.log: http_request.on_abort
switch:
  - platform: template
    id: ${name}force_update
    name: Force Update
    optimistic: true

button:
  - platform: safe_mode
    id: ${name}_safe_mode
    name: Restart in Safe Mode button
  - platform: template
    name: Firmware update
    id: ${name}_firmware_update
    on_press:
      then:
        - ota_http_request.flash:
            md5_url: https://myhome.sfdp.net/local/${name}/firmware.md5
            url: https://myhome.sfdp.net/local/${name}/firmware.bin
            #force_update: !lambda "return id(${name}force_update).state;"
        - logger.log: "This message should be not displayed (reboot)"

Can you please be so kind to advice me on what I am doing wrong.
Thanks for your attention.
Ngoc!

oarcher · 2024-06-05T21:06:46Z

@paulsteigel , the force_update option has been removed.

For HTTP server did not respond (error -1), I think that you have bad network configuration.
Can you check with a simple http_request ?

http_request:
  id: http_data


button:
  - platform: template
    name: "http_request test"
    on_press:
      - http_request.get:
          url: https://myhome.sfdp.net/local/${name}/firmware.md5
          verify_ssl: false
          on_response:
            then:
              - logger.log:
                  format: 'Response status: %d, Duration: %u ms'
                  args:
                    - status_code
                    - duration_ms
              - lambda: |-
                  ESP_LOGI("test", "content: %s", id(http_data).get_string());

Also try with a public server url, like https://api.ipify.org/?format=text

paulsteigel · 2024-06-06T15:32:24Z

@paulsteigel , the force_update option has been removed.

For HTTP server did not respond (error -1), I think that you have bad network configuration. Can you check with a simple http_request ?

http_request:
  id: http_data


button:
  - platform: template
    name: "http_request test"
    on_press:
      - http_request.get:
          url: https://myhome.sfdp.net/local/${name}/firmware.md5
          verify_ssl: false
          on_response:
            then:
              - logger.log:
                  format: 'Response status: %d, Duration: %u ms'
                  args:
                    - status_code
                    - duration_ms
              - lambda: |-
                  ESP_LOGI("test", "content: %s", id(http_data).get_string());

Also try with a public server url, like https://api.ipify.org/?format=text

Many thanks to @oarcher
I will test tomorrow! One thing specifically, my HA often restart since installing the Esphome Dev and the Ota_http component! I tried checking the system log but the reason is not known yet!
I will check problem in detail during this weekend!
Best
Ngoc

tomatensaus · 2024-06-06T18:32:18Z

@paulsteigel

Can you please be so kind to advice me on what I am doing wrong.
Thanks for your attention.

If you specify a static IP be sure to also include the nameserver entry in your network configuration. I once spent a whole day trying to figure out why I could work from an IP and not to my domain. I remember the error message was very confusing but I cannot recall if this is the same case

probot-esphome bot added needs-tests new-integration integration: ota_http needs-docs needs-codeowners labels Oct 22, 2023

probot-esphome bot added has-tests and removed needs-codeowners needs-tests labels Oct 22, 2023

oarcher mentioned this pull request Oct 23, 2023

OTA Update via HTTP Request documentation esphome/esphome-docs#3291

Merged

3 tasks

probot-esphome bot removed the needs-docs label Oct 23, 2023

oarcher mentioned this pull request Oct 27, 2023

Read/Write bootloader, partition table and any partition via OTA #5535

Open

11 tasks

oarcher mentioned this pull request Nov 12, 2023

OTA update via HTTP esphome/feature-requests#801

Closed

oarcher marked this pull request as draft November 16, 2023 21:44

oarcher mentioned this pull request Nov 17, 2023

ota_http crash if out of memory. Implement safe mode ? oarcher/piotech#15

Closed

oarcher mentioned this pull request Dec 7, 2023

Split socket code from OTA into extra componnent angelnu/esphome-1#1

Closed

oarcher marked this pull request as ready for review December 7, 2023 23:20

kbx81 and others added 8 commits May 26, 2024 23:11

Update esphome/components/http_request/ota/__init__.py with a walrus

310751c

Co-authored-by: Jesse Hills <3060199+jesserockz@users.noreply.github.com>

Update esphome/components/http_request/ota/__init__.py

6536c15

Co-authored-by: Jesse Hills <3060199+jesserockz@users.noreply.github.com>

Missed removing a force_update

0893222

Co-authored-by: Jesse Hills <3060199+jesserockz@users.noreply.github.com>

Implemented suggestions from review

1d8a3cb

Fix test + RP2040

5d4b580

Merge branch 'dev' into dev

4f71506

Use helper class to manage watchdog adjustments

f6cdf40

Merge branch 'dev' into dev

132cd22

kbx81 marked this pull request as ready for review May 29, 2024 07:11

kbx81 added 3 commits May 29, 2024 18:22

A bit more tidying up

6930a49

Move watchdog bits

3391710

Merge branch 'dev' into dev

366dc56

randybb mentioned this pull request May 30, 2024

sftp "flash on boot" esphome/feature-requests#2742

Closed

Merge branch 'dev' into dev

780d54f

kbx81 changed the title ~~OTA via HTTP request update platform~~ [ota] http_request update platform Jun 1, 2024

kbx81 added 6 commits June 3, 2024 13:51

Merge branch 'dev' into dev

bab656d

Merge branch 'dev' into dev

6c6a7eb

Tidy up watchdog bits

3ecb1cd

Add some NOLINTs

18b00a4

Use verify_ssl, related validation tweaks

73aa400

Fix tests, RP2040 validation

5220be8

Add define

83d6643

probot-esphome bot added the core label Jun 5, 2024

Clean up some headers

5a98daa

jesserockz approved these changes Jun 6, 2024

View reviewed changes

jesserockz merged commit 8ef4aaa into esphome:dev Jun 6, 2024
56 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ota] http_request update platform #5586

[ota] http_request update platform #5586

oarcher commented Oct 22, 2023 •

edited

probot-esphome bot commented Oct 22, 2023

oarcher commented Oct 23, 2023 •

edited

nerijus commented Oct 25, 2023

oarcher commented Oct 26, 2023

oarcher commented Oct 27, 2023 •

edited

nerijus commented Oct 27, 2023

angelnu commented Nov 19, 2023 •

edited

Alphaemef commented Dec 23, 2023

codecov-commenter commented Jan 11, 2024 •

edited

kevireilly commented Feb 13, 2024

paulsteigel commented Feb 21, 2024 •

edited

oarcher commented Feb 21, 2024

oarcher commented Feb 21, 2024

paulsteigel commented Feb 22, 2024 •

edited

paulsteigel commented Jun 5, 2024 •

edited

oarcher commented Jun 5, 2024 •

edited

paulsteigel commented Jun 6, 2024

tomatensaus commented Jun 6, 2024

[ota] http_request update platform #5586

[ota] http_request update platform #5586

Conversation

oarcher commented Oct 22, 2023 • edited

What does this implement/fix?

Types of changes

Test Environment

Http/Https support matrix

Example entry for config.yaml:

Example config.yaml

Checklist:

probot-esphome bot commented Oct 22, 2023

oarcher commented Oct 23, 2023 • edited

nerijus commented Oct 25, 2023

oarcher commented Oct 26, 2023

oarcher commented Oct 27, 2023 • edited

nerijus commented Oct 27, 2023

angelnu commented Nov 19, 2023 • edited

Alphaemef commented Dec 23, 2023

codecov-commenter commented Jan 11, 2024 • edited

Codecov Report

kevireilly commented Feb 13, 2024

paulsteigel commented Feb 21, 2024 • edited

oarcher commented Feb 21, 2024

oarcher commented Feb 21, 2024

paulsteigel commented Feb 22, 2024 • edited

paulsteigel commented Jun 5, 2024 • edited

oarcher commented Jun 5, 2024 • edited

paulsteigel commented Jun 6, 2024

tomatensaus commented Jun 6, 2024

oarcher commented Oct 22, 2023 •

edited

Example entry for `config.yaml`:

oarcher commented Oct 23, 2023 •

edited

oarcher commented Oct 27, 2023 •

edited

angelnu commented Nov 19, 2023 •

edited

codecov-commenter commented Jan 11, 2024 •

edited

paulsteigel commented Feb 21, 2024 •

edited

paulsteigel commented Feb 22, 2024 •

edited

paulsteigel commented Jun 5, 2024 •

edited

oarcher commented Jun 5, 2024 •

edited