Proposal v4.0 b1 (HTTP/2+3, OS Trust Store, Custom DNS, OCSP, ...) #1531

Ousret · 2023-10-03T12:30:27Z

This PR showcases how HTTPie could evolve outside of Requests.

Niquests is supposed to be a (mostly) compatible fork of Requests.

Try this preview:

$ pip install "git+https://github.com/Ousret/HTTPie.git@feature-tryout-niquests" -U

Here are the biggest pros of this:

OS truststore by default, no more certifi!
Object-oriented headers. Could bring additional features!
Fully type-annotated!
HTTP/3 over QUIC.
HTTP/2 by default.
Exit Python http.client! in favor of h11.
Timeout by default.
Inspect peer certificate, HTTP version, TLS version, cipher, and so on via hook/callback before a request is sent.
All the features you expose are available in all three protocols.
Python 3.7+, no sacrifice needed.
Encrypted DNS support. w/ DNS-over-HTTPS, DNS-over-QUIC, DNS-over-TLS and plain DNS-over-UDP.

Obviously, cons:

Stricter on emitted requests per RFCs
Young project but based on solid bases, knowledge, and experiences.
Need to publish new packages to distro. Easy but time-consuming
Exit pyopenssl, not supported. more of a pro to me
Require major bump? Could be. Should be.

Complete list of changes in the fork: https://github.com/jawah/niquests/blob/main/HISTORY.md

4.0.0.b1 (unreleased)

Make it possible to unset the User-Agent, and Accept-Encoding headers. (#1502)
Dependency on requests was changed in favor of compatible niquests. (#1531)
Added support for HTTP/2, and HTTP/3 protocols. (#523) (#692) (#1531)
Added request metadata for the TLS certificate, negotiated version with cipher, the revocation status and the remote peer IP address. (#1495) (#1023) (#826) (#1531)
Added support to load the operating system trust store for the peer certificate validation. (#480) (#1531)
Added detailed timings in response metadata with DNS resolution, established, TLS handshake, and request sending delays. (#1023) (#1531)
Added support for using alternative DNS resolver using --resolver. DNS over HTTPS, DNS over TLS, DNS over QUIC, and DNS over UDP are accepted. (#99) (#1531)
Added support for binding to a specific network adapter with --interface. (#1422) (#1531)
Added support for specifying the local port with --local-port. (#1456) (#1531)
Added support for forcing either IPv4 or IPv6 to reach the remote HTTP server with -6 or -4. (#94) (#1531)
Removed support for pyopenssl. (#1531)
Removed support for dead SSL protocols < TLS 1.0 (e.g. sslv3) as per pyopenssl removal. (#1531)
Dropped dependency on requests_toolbelt in favor of directly including MultipartEncoder into HTTPie due to its direct dependency to requests. (#1531)
Dropped dependency on multidict in favor of implementing an internal one due to often missing pre-built wheels. (#1522) (#1531)
Fixed the case when multiple headers where concatenated in the response output. (#1413) (#1531)
Fixed an edge case where HTTPie could be lead to believe data was passed in stdin, thus sending a POST by default. (#1551) (#1531)
This fix has the particularity to consider 0 byte long stdin buffer as absent stdin. Empty stdin buffer will be ignored.
Slightly improved performance while downloading by setting chunk size to -1 to retrieve packets as they arrive. (#1531)
Added support for using the system trust store to retrieve root CAs for verifying TLS certificates. (#1531)
Removed support for keeping the original casing of HTTP headers. This come from an outer constraint by newer protocols, namely HTTP/2+ that normalize header keys by default.
From the HTTPie user perspective, they are "prettified" on the output by default. e.g. "x-hello-world" is displayed as "X-Hello-World".
Fixed multipart form data having filename not rfc2231 compliant when name contain non-ascii characters. (#1401)
Fixed issue where the configuration directory was not created at runtime that made the update fetcher run everytime. (#1527)
Fixed cookie persistence in HTTPie session when targeting localhost. They were dropped due to the standard library. (#1527)
Fixed downloader when trying to fetch compressed content. The process will no longer exit with the "Incomplete download" error. (#1554) (#423) (#1527)
Added automated resolution of hosts ending with .localhost to the default loopback address. (#1458) (#1527)

Existing plugins are expected to work without any changes. The only caveat would be that certain plugin explicitly require requests.
Future contributions may be made in order to relax the constraints where applicable.

tests/test_json.py

codecov-commenter · 2023-10-13T16:39:04Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 0.00%. Comparing base (4d7d6b6) to head (7cd6579).
Report is 370 commits behind head on master.

❗ Current head 7cd6579 differs from pull request most recent head bb3583c

Please upload reports for the commit bb3583c to get more accurate results.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #1531       +/-   ##
==========================================
- Coverage   97.28%       0   -97.29%     
==========================================
  Files          67       0       -67     
  Lines        4235       0     -4235     
==========================================
- Hits         4120       0     -4120     
+ Misses        115       0      -115

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Ousret · 2023-10-23T11:59:01Z

So far, I identified this:

Ousret · 2023-10-23T12:08:46Z

We can add the following arguments and document them:

--disable-http2
--disable-http3
--http3, e.g. force trying it.

Ousret · 2023-11-16T06:47:26Z

I've added and documented it.

--disable-http2
--disable-http3
--http3, e.g. force trying it.

Ideally, the connection information (TLS, Cipher, Peer/Issuer Certificate, and OCSP) should be in the request metadata and with a lexer instruction.

Ousret · 2023-11-27T19:02:12Z

Now, the result is pleasant to the eye.
I have implemented the metadata for Request and the result is as follows:

Niquests now support tracking upload progress, this can also be done here. It should be straightforward enough.

dwt · 2023-12-20T13:33:06Z

Damn this looks so tantalizing. I'd love for this to get packaged as a tryout / beta. Something installable via e.g. pip install httpie[niquest] perhaps?

markstos · 2024-05-20T13:04:01Z

Looks like it could be a big step forward if @jkbrzt approves it. Last week I wanted to test resolving over IPv4 vs IPv6. I found that the released version could not help with this task, but it worked perfectly in the fork, with the same -4 and -6 flags that dig uses.

non https server were expected to receive "secure" cookies...? this seems to be a bug that lied in Requests for quite some time.

ducaale · 2024-05-23T04:15:58Z

Excited to see HTTP/2 and HTTP/3 soon landing in HTTPie 🎉

I may be missing something here, but does that imply that the client has no way to force http1 2 or 3 according to its wishes?

There's many possibilities, the only missing piece is "force http2" or "disable http1".
"--disable-http2" HTTP/1.1 or HTTP/3
"--disable-http2 --disable-http3" HTTP/1.1
"--disable-http3" HTTP/1.1 or HTTP/2
"--http3" HTTP/3

I was wondering if we could have an equivalent to cURL's http-version flags i.e --http1.0,--http1.1,--http2,--http2-prior-knowledge,--http3? Alternatively, it could be a --http-version flag that accepts either 1.0, 1.1, 2, 2-prior-knowledge or 3

Ousret · 2024-05-23T04:19:25Z

Excited to see HTTP/2 and HTTP/3 soon landing in HTTPie 🎉

Likewise!

I was wondering if we could have an equivalent to cURL's http-version flags

I first need to add a feature into Niquests, which isn't top priority right now.
But it is planed, will be in a future minor. Provided everything goes smooth with this.

regards,

close httpie#1458

docs/README.md

…to feature-tryout-niquests

janbrasna

docs typos:

CHANGELOG.md

Co-authored-by: Jan Brasna <1784648+janbrasna@users.noreply.github.com>

janbrasna

A couple of questions regarding how to surface some of the functionality and constraints to the users:

extras/man/http.1

extras/man/https.1

janbrasna · 2024-06-06T13:04:20Z

extras/man/http.1

@@ -570,9 +570,10 @@ installation (only the supported ones are shown here).


 A string in the OpenSSL cipher list format.
+tls1.3 ciphers are always present regardless of your cipher list.


This may need a bit more context or outsider-friendly rewording. WDYT?

Suggested change

tls1.3 ciphers are always present regardless of your cipher list.

Applies only for TLS v1.2 and lower. TLS v1.3 connections always use the appropriate cipher suites available for that protocol version, regardless of this setting.

see https://docs.python.org/3/library/ssl.html#ssl.SSLContext.set_ciphers

we can't do anything about it. unless we're using http3 over quic.

Shortstory: Python hardcode it.

I'm not challenging anything from that. Most APIs are only handling tls12 ciphers, and don't bother with lower-level ctxs to set tls13 suites, esp. in OpenSSL world. — I'm only raising the need to explain that to consumers not familiar with the distinction in the new protocol RFC.

extras/man/https.1

httpie/cli/definition.py

httpie/ssl_.py

Ousret mentioned this pull request Oct 3, 2023

Add support for HTTP/2, HTTP/3, System CA, OCSP Revocation #1512

Open

Ousret force-pushed the feature-tryout-niquests branch from 46f030c to 16ccc1b Compare October 3, 2023 12:35

Ousret mentioned this pull request Oct 3, 2023

Add HTTP/3 to mapping in property version (HTTPResponse) #1513

Closed

Ousret force-pushed the feature-tryout-niquests branch from 16ccc1b to 741017e Compare October 11, 2023 18:44

This comment was marked as resolved.

Sign in to view

Ousret commented Oct 11, 2023

View reviewed changes

tests/test_json.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

Ousret force-pushed the feature-tryout-niquests branch 3 times, most recently from faf9aef to c12f000 Compare October 13, 2023 16:37

This comment was marked as outdated.

Sign in to view

This comment was marked as resolved.

Sign in to view

Ousret force-pushed the feature-tryout-niquests branch 4 times, most recently from 52653e4 to 247b382 Compare October 16, 2023 05:45

Ousret force-pushed the feature-tryout-niquests branch 2 times, most recently from 0b4eebb to 7dd1152 Compare October 23, 2023 10:56

Ousret force-pushed the feature-tryout-niquests branch from 7dd1152 to 7257a74 Compare November 16, 2023 06:45

Ousret changed the title ~~⚗️ Try compatible fork Niquests to supercharge HTTPie~~ ⚗️ Add support for HTTP/2, HTTP/3, System CA, OCSP Revocation, Extended Conn Info Nov 16, 2023

Ousret force-pushed the feature-tryout-niquests branch from 7257a74 to 4c246e4 Compare November 27, 2023 18:59

Ousret force-pushed the feature-tryout-niquests branch from 4c246e4 to f034379 Compare November 27, 2023 19:06

Ousret changed the title ~~⚗️ Add support for HTTP/2, HTTP/3, System CA, OCSP Revocation, Extended Conn Info~~ Add support for HTTP/2, HTTP/3, System CA, OCSP Revocation, Extended Conn Info Dec 10, 2023

Ousret force-pushed the feature-tryout-niquests branch from f034379 to df42bba Compare January 1, 2024 18:22

Tweak

007234f

This was referenced May 21, 2024

Cookies with "Domain=localhost" aren't getting stored in session file #602

Open

Change insecure SSL/TLS to deprecated #962

Open

fix test_secure_cookies_on_localhost case

71ab43e

non https server were expected to receive "secure" cookies...? this seems to be a bug that lied in Requests for quite some time.

This was referenced May 23, 2024

httpie -d does not work with gzip compressed content #1554

Open

--download reports an "incomplete download" with Content-Encoding: gzip and content-length. #423

Open

fix downloader with compressed content httpie#1554 httpie#423

d9db5a2

Ousret force-pushed the feature-tryout-niquests branch from dec765c to d9db5a2 Compare May 23, 2024 04:13

Ousret mentioned this pull request May 23, 2024

Deprecate PROTOCOL_SSLv23 #1400

Open

add support for resolving ".localhost" domains

ec02b1c

close httpie#1458

Ousret mentioned this pull request May 24, 2024

Feature Request: Support special-use domain names like localhost. #1458

Open

1 task

Changelog tweaks

2583671

janbrasna reviewed May 25, 2024

View reviewed changes

docs/README.md Outdated Show resolved Hide resolved

Ousret added 2 commits May 26, 2024 05:52

adjust docs around tls version cli command

523faa6

Merge branch 'feature-tryout-niquests' of github.com:Ousret/httpie in…

d200787

…to feature-tryout-niquests

Ousret force-pushed the feature-tryout-niquests branch from 94c960e to 7cd6579 Compare May 26, 2024 09:02

improve support --ssl=[...] setting

e3cbc7f

Ousret force-pushed the feature-tryout-niquests branch from 7cd6579 to e3cbc7f Compare May 26, 2024 09:11

jkbrzt added 5 commits June 3, 2024 13:49

Docs wip

78aa25d

Add cd_clean_tmp_dir() test utility context manager

d238725

Cleanup imports

4747669

Tests

b3f9e01

Clean

bb3583c

janbrasna suggested changes Jun 6, 2024

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

CHANGELOG.md Outdated Show resolved Hide resolved

jkbrzt and others added 2 commits June 6, 2024 06:08

Update CHANGELOG.md

77bbf8c

Co-authored-by: Jan Brasna <1784648+janbrasna@users.noreply.github.com>

Update CHANGELOG.md

128d952

Co-authored-by: Jan Brasna <1784648+janbrasna@users.noreply.github.com>

janbrasna reviewed Jun 6, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Proposal v4.0 b1 (HTTP/2+3, OS Trust Store, Custom DNS, OCSP, ...) #1531

Proposal v4.0 b1 (HTTP/2+3, OS Trust Store, Custom DNS, OCSP, ...) #1531

Ousret commented Oct 3, 2023 •

edited

This comment was marked as resolved.

This comment was marked as resolved.

codecov-commenter commented Oct 13, 2023 •

edited

This comment was marked as outdated.

This comment was marked as resolved.

Ousret commented Oct 23, 2023 •

edited

Ousret commented Oct 23, 2023

Ousret commented Nov 16, 2023 •

edited

Ousret commented Nov 27, 2023

dwt commented Dec 20, 2023

markstos commented May 20, 2024

ducaale commented May 23, 2024

Ousret commented May 23, 2024

janbrasna left a comment

janbrasna left a comment

janbrasna Jun 6, 2024

Ousret Jun 6, 2024

Ousret Jun 6, 2024

janbrasna Jun 6, 2024

		@@ -570,9 +570,10 @@ installation (only the supported ones are shown here).


		A string in the OpenSSL cipher list format.
		tls1.3 ciphers are always present regardless of your cipher list.

	tls1.3 ciphers are always present regardless of your cipher list.
	Applies only for TLS v1.2 and lower. TLS v1.3 connections always use the appropriate cipher suites available for that protocol version, regardless of this setting.

Proposal v4.0 b1 (HTTP/2+3, OS Trust Store, Custom DNS, OCSP, ...) #1531

Are you sure you want to change the base?

Proposal v4.0 b1 (HTTP/2+3, OS Trust Store, Custom DNS, OCSP, ...) #1531

Conversation

Ousret commented Oct 3, 2023 • edited

4.0.0.b1 (unreleased)

This comment was marked as resolved.

This comment was marked as resolved.

codecov-commenter commented Oct 13, 2023 • edited

Codecov Report

This comment was marked as outdated.

This comment was marked as resolved.

Ousret commented Oct 23, 2023 • edited

Ousret commented Oct 23, 2023

Ousret commented Nov 16, 2023 • edited

Ousret commented Nov 27, 2023

dwt commented Dec 20, 2023

markstos commented May 20, 2024

ducaale commented May 23, 2024

Ousret commented May 23, 2024

janbrasna left a comment

Choose a reason for hiding this comment

janbrasna left a comment

Choose a reason for hiding this comment

janbrasna Jun 6, 2024

Choose a reason for hiding this comment

Ousret Jun 6, 2024

Choose a reason for hiding this comment

Ousret Jun 6, 2024

Choose a reason for hiding this comment

janbrasna Jun 6, 2024

Choose a reason for hiding this comment

Ousret commented Oct 3, 2023 •

edited

codecov-commenter commented Oct 13, 2023 •

edited

Ousret commented Oct 23, 2023 •

edited

Ousret commented Nov 16, 2023 •

edited